[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 07:01:18 dist-test-slave-ncp9 krb5kdc[21855](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 07:01:18 dist-test-slave-ncp9 krb5kdc[21855](info): set up 2 sockets
Aug 07 07:01:18 dist-test-slave-ncp9 krb5kdc[21855](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260807 07:01:19.339479 21836 mini_kdc.cc:121] Time spent starting KDC: real 1.171s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Aug 07 07:01:19 dist-test-slave-ncp9 krb5kdc[21855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T07:01:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:01:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
I20260807 07:01:19.532833 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:43965
--webserver_interface=127.21.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:38087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:43965
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:01:19.888101 21872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:01:19.888455 21872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:01:19.888602 21872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:01:19.895622 21872 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:01:19.895714 21872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:01:19.895783 21872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:01:19.895864 21872 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:01:19.895925 21872 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:01:19.905776 21872 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:38087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:43965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:43965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:01:19.907274 21872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:01:19.908721 21872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:01:19.923486 21883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:19.923671 21886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:19.923851 21882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:01:19.924741 21872 server_base.cc:1034] running on GCE node
I20260807 07:01:19.925611 21872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:01:19.927505 21872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:01:19.928716 21872 hybrid_clock.cc:648] HybridClock initialized: now 1786086079928664 us; error 44 us; skew 500 ppm
I20260807 07:01:19.929128 21872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:01:19 dist-test-slave-ncp9 krb5kdc[21855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086079, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:01:19.936235 21872 init.cc:377] Logged in from keytab as kudu/127.21.83.62@KRBTEST.COM (short username kudu)
I20260807 07:01:19.937878 21872 webserver.cc:460] Webserver started at http://127.21.83.62:41423/ using document root <none> and password file <none>
I20260807 07:01:19.938736 21872 fs_manager.cc:359] Metadata directory not provided
I20260807 07:01:19.938856 21872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:01:19.939145 21872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:01:19.942586 21872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "c9843ddf80ac4b53aaadf97dbadb6350"
format_stamp: "Formatted at 2026-08-07 07:01:19 on dist-test-slave-ncp9"
server_key: "e78ca63092699fc97ca2a2348011c884"
server_key_iv: "e677f9f157f0f56d6e8ae90b5eddd00a"
server_key_version: "encryptionkey@0"
I20260807 07:01:19.943310 21872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "c9843ddf80ac4b53aaadf97dbadb6350"
format_stamp: "Formatted at 2026-08-07 07:01:19 on dist-test-slave-ncp9"
server_key: "e78ca63092699fc97ca2a2348011c884"
server_key_iv: "e677f9f157f0f56d6e8ae90b5eddd00a"
server_key_version: "encryptionkey@0"
I20260807 07:01:19.948740 21872 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 07:01:19.952078 21899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:19.953414 21872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 07:01:19.953591 21872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "c9843ddf80ac4b53aaadf97dbadb6350"
format_stamp: "Formatted at 2026-08-07 07:01:19 on dist-test-slave-ncp9"
server_key: "e78ca63092699fc97ca2a2348011c884"
server_key_iv: "e677f9f157f0f56d6e8ae90b5eddd00a"
server_key_version: "encryptionkey@0"
I20260807 07:01:19.953774 21872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:01:19.971951 21872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:01:19.976408 21872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:01:19.976703 21872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:01:19.994474 21872 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.62:43965
I20260807 07:01:19.994537 21976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.62:43965 every 8 connection(s)
I20260807 07:01:19.996074 21872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:01:20.000109 21977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:01:20.000138 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 21872
I20260807 07:01:20.000437 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:01:20.002245 21836 external_mini_cluster.cc:1428] Setting key cda68c1ab843b5e35688881eaa3be2ae
I20260807 07:01:20.010485 21977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9843ddf80ac4b53aaadf97dbadb6350: Bootstrap starting.
I20260807 07:01:20.013454 21977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9843ddf80ac4b53aaadf97dbadb6350: Neither blocks nor log segments found. Creating new log.
I20260807 07:01:20.014421 21977 log.cc:824] T 00000000000000000000000000000000 P c9843ddf80ac4b53aaadf97dbadb6350: Log is configured to *not* fsync() on all Append() calls
I20260807 07:01:20.017206 21977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9843ddf80ac4b53aaadf97dbadb6350: No bootstrap required, opened a new log
I20260807 07:01:20.026755 21977 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9843ddf80ac4b53aaadf97dbadb6350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9843ddf80ac4b53aaadf97dbadb6350" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 43965 } }
I20260807 07:01:20.027216 21977 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9843ddf80ac4b53aaadf97dbadb6350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:01:20.027330 21977 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9843ddf80ac4b53aaadf97dbadb6350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9843ddf80ac4b53aaadf97dbadb6350, State: Initialized, Role: FOLLOWER
I20260807 07:01:20.028079 21977 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9843ddf80ac4b53aaadf97dbadb6350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9843ddf80ac4b53aaadf97dbadb6350" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 43965 } }
I20260807 07:01:20.028335 21977 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9843ddf80ac4b53aaadf97dbadb6350 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:01:20.028476 21977 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9843ddf80ac4b53aaadf97dbadb6350 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:01:20.028618 21977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9843ddf80ac4b53aaadf97dbadb6350 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:01:20.030773 21977 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9843ddf80ac4b53aaadf97dbadb6350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9843ddf80ac4b53aaadf97dbadb6350" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 43965 } }
I20260807 07:01:20.031361 21977 leader_election.cc:302] T 00000000000000000000000000000000 P c9843ddf80ac4b53aaadf97dbadb6350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9843ddf80ac4b53aaadf97dbadb6350; no voters:
I20260807 07:01:20.031935 21977 leader_election.cc:288] T 00000000000000000000000000000000 P c9843ddf80ac4b53aaadf97dbadb6350 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 07:01:20.032025 21982 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9843ddf80ac4b53aaadf97dbadb6350 [term 1 FOLLOWER]: Leader election won for term 1
Aug 07 07:01:20 dist-test-slave-ncp9 krb5kdc[21855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:01:20.034210 21982 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9843ddf80ac4b53aaadf97dbadb6350 [term 1 LEADER]: Becoming Leader. State: Replica: c9843ddf80ac4b53aaadf97dbadb6350, State: Running, Role: LEADER
I20260807 07:01:20.034217 21977 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9843ddf80ac4b53aaadf97dbadb6350 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:01:20.034878 21982 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9843ddf80ac4b53aaadf97dbadb6350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9843ddf80ac4b53aaadf97dbadb6350" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 43965 } }
I20260807 07:01:20.040766 21985 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9843ddf80ac4b53aaadf97dbadb6350 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9843ddf80ac4b53aaadf97dbadb6350. Latest consensus state: current_term: 1 leader_uuid: "c9843ddf80ac4b53aaadf97dbadb6350" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9843ddf80ac4b53aaadf97dbadb6350" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 43965 } } }
I20260807 07:01:20.041128 21985 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9843ddf80ac4b53aaadf97dbadb6350 [sys.catalog]: This master's current role is: LEADER
I20260807 07:01:20.044534 21984 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9843ddf80ac4b53aaadf97dbadb6350 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9843ddf80ac4b53aaadf97dbadb6350" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9843ddf80ac4b53aaadf97dbadb6350" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 43965 } } }
I20260807 07:01:20.044844 21984 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9843ddf80ac4b53aaadf97dbadb6350 [sys.catalog]: This master's current role is: LEADER
I20260807 07:01:20.045039 21979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:20.005978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42160 (local address 127.21.83.62:43965)
0807 07:01:20.006883 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:20.006897 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:20.010331 (+ 3434us) server_negotiation.cc:407] Connection header received
0807 07:01:20.019420 (+ 9089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:20.019446 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:20.019975 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:20.020385 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:20.023056 (+ 2671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:20.025105 (+ 2049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:20.026786 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:20.027790 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:20.027973 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:20.034087 (+ 6114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:20.034100 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:20.034120 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:20.034179 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:20.038619 (+ 4440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:20.039635 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:20.039643 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:20.039649 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:20.039871 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:20.040442 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:20.040447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:20.040450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:20.041368 (+ 918us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:20.041818 (+ 450us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:20.043105 (+ 1287us) server_negotiation.cc:299] Negotiation successful
0807 07:01:20.043520 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":405,"threads_started":1}
W20260807 07:01:20.051962 22010 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c9843ddf80ac4b53aaadf97dbadb6350: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 07:01:20.052098 22010 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 07:01:20.052793 22000 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:01:20.054847 22000 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:01:20.064253 22000 catalog_manager.cc:1347] Generated new cluster ID: 1ea5e9b908d94be59019719a21be5427
I20260807 07:01:20.064388 22000 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:01:20.085700 22000 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:01:20.087011 22000 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:01:20.094261 22000 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9843ddf80ac4b53aaadf97dbadb6350: Generated new TSK 0
I20260807 07:01:20.095021 22000 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.83.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab.
Entry for principal kudu/127.21.83.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab.
I20260807 07:01:20.141597 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.1:0
--local_ip_for_outbound_sockets=127.21.83.1
--webserver_interface=127.21.83.1
--webserver_port=0
--tserver_master_addrs=127.21.83.62:43965
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab
--principal=kudu/127.21.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:38087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:01:20.477138 22019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:01:20.477466 22019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:01:20.477595 22019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:01:20.485224 22019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:01:20.485361 22019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:01:20.485525 22019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.1
I20260807 07:01:20.495832 22019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:38087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab
--principal=kudu/127.21.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.83.1
--webserver_port=0
--tserver_master_addrs=127.21.83.62:43965
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.1
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:01:20.497262 22019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:01:20.498636 22019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:01:20.512531 22027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:20.512689 22031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:20.512805 22025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:01:20.513477 22019 server_base.cc:1034] running on GCE node
I20260807 07:01:20.514107 22019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:01:20.515101 22019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:01:20.516330 22019 hybrid_clock.cc:648] HybridClock initialized: now 1786086080516262 us; error 50 us; skew 500 ppm
I20260807 07:01:20.516650 22019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:01:20 dist-test-slave-ncp9 krb5kdc[21855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086080, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:01:20.519838 22019 init.cc:377] Logged in from keytab as kudu/127.21.83.1@KRBTEST.COM (short username kudu)
I20260807 07:01:20.521447 22019 webserver.cc:460] Webserver started at http://127.21.83.1:37631/ using document root <none> and password file <none>
I20260807 07:01:20.522143 22019 fs_manager.cc:359] Metadata directory not provided
I20260807 07:01:20.522306 22019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:01:20.522630 22019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:01:20.525627 22019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/ts-0/data/instance:
uuid: "dab055bfc11046fcb0dc213e209c1a69"
format_stamp: "Formatted at 2026-08-07 07:01:20 on dist-test-slave-ncp9"
server_key: "6550e344e9e38d1f09e05b4fe43fe061"
server_key_iv: "70abb5635b5fad1a12dbacac0bb31d27"
server_key_version: "encryptionkey@0"
I20260807 07:01:20.526383 22019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/ts-0/wal/instance:
uuid: "dab055bfc11046fcb0dc213e209c1a69"
format_stamp: "Formatted at 2026-08-07 07:01:20 on dist-test-slave-ncp9"
server_key: "6550e344e9e38d1f09e05b4fe43fe061"
server_key_iv: "70abb5635b5fad1a12dbacac0bb31d27"
server_key_version: "encryptionkey@0"
I20260807 07:01:20.531426 22019 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 07:01:20.534886 22046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:20.536218 22019 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260807 07:01:20.536365 22019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/ts-0/wal
uuid: "dab055bfc11046fcb0dc213e209c1a69"
format_stamp: "Formatted at 2026-08-07 07:01:20 on dist-test-slave-ncp9"
server_key: "6550e344e9e38d1f09e05b4fe43fe061"
server_key_iv: "70abb5635b5fad1a12dbacac0bb31d27"
server_key_version: "encryptionkey@0"
I20260807 07:01:20.536547 22019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:01:20.554360 22019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:01:20.559219 22019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:01:20.559512 22019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:01:20.560583 22019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:01:20.562178 22019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:01:20.562285 22019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:20.562398 22019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:01:20.562474 22019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:20.594738 22019 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.1:35813
I20260807 07:01:20.594792 22218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.1:35813 every 8 connection(s)
I20260807 07:01:20.595979 22019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
I20260807 07:01:20.601709 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 22019
I20260807 07:01:20.601962 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/ts-0/wal/instance
I20260807 07:01:20.602624 21836 external_mini_cluster.cc:1428] Setting key 4f7ac96ec3c9a73523ca7165ce15ca4b
Aug 07 07:01:20 dist-test-slave-ncp9 krb5kdc[21855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086080, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.1@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:01:20.617050 22222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:20.598462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:54151 (local address 127.21.83.62:43965)
0807 07:01:20.598922 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:20.598928 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:20.599855 (+ 927us) server_negotiation.cc:407] Connection header received
0807 07:01:20.601680 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:20.601686 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:20.601773 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:20.601901 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:20.603442 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:20.604296 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:20.605918 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:20.606552 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:20.606684 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:20.610067 (+ 3383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:20.610074 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:20.610079 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:20.610124 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:20.613365 (+ 3241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:20.614268 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:20.614273 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:20.614276 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:20.614431 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:20.614932 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:20.614936 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:20.614939 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:20.615318 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:20.615467 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:20.616523 (+ 1056us) server_negotiation.cc:299] Negotiation successful
0807 07:01:20.616734 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":222,"threads_started":1}
I20260807 07:01:20.618443 22223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:20.598981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:43965 (local address 127.21.83.1:54151)
0807 07:01:20.599697 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:20.599739 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:20.601418 (+ 1679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:20.602118 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:20.602126 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:20.602661 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:01:20.603252 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:20.603278 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:20.604450 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:20.604455 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:20.605765 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:20.605777 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:20.606768 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:20.606773 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:20.606973 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:20.607614 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:01:20.607637 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:01:20.609831 (+ 2194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:01:20.613537 (+ 3706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:20.613612 (+ 75us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:20.613641 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:20.614123 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:20.614570 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:20.614574 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:20.614576 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:20.614808 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:20.615459 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:01:20.615468 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:01:20.616069 (+ 601us) client_negotiation.cc:769] Sending connection context
0807 07:01:20.616405 (+ 336us) client_negotiation.cc:240] Negotiation successful
0807 07:01:20.617038 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":282,"threads_started":1}
I20260807 07:01:20.620249 22219 heartbeater.cc:347] Connected to a master server at 127.21.83.62:43965
I20260807 07:01:20.620579 22219 heartbeater.cc:464] Registering TS with master...
I20260807 07:01:20.621435 22219 heartbeater.cc:511] Master 127.21.83.62:43965 requested a full tablet report, sending...
I20260807 07:01:20.623844 21924 ts_manager.cc:194] Registered new tserver with Master: dab055bfc11046fcb0dc213e209c1a69 (127.21.83.1:35813)
I20260807 07:01:20.625406 21924 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.1@KRBTEST.COM'} at 127.21.83.1:54151
WARNING: no policy specified for kudu/127.21.83.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab.
Entry for principal kudu/127.21.83.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab.
I20260807 07:01:20.656298 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.2:0
--local_ip_for_outbound_sockets=127.21.83.2
--webserver_interface=127.21.83.2
--webserver_port=0
--tserver_master_addrs=127.21.83.62:43965
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab
--principal=kudu/127.21.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:38087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:01:20.994632 22229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:01:20.994992 22229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:01:20.995147 22229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:01:21.002753 22229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:01:21.002878 22229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:01:21.003042 22229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.2
I20260807 07:01:21.012825 22229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:38087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab
--principal=kudu/127.21.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.83.2
--webserver_port=0
--tserver_master_addrs=127.21.83.62:43965
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.2
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:01:21.014165 22229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:01:21.015513 22229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:01:21.028817 22238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:21.028862 22239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:21.028746 22242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:01:21.029625 22229 server_base.cc:1034] running on GCE node
I20260807 07:01:21.030251 22229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:01:21.031235 22229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:01:21.032466 22229 hybrid_clock.cc:648] HybridClock initialized: now 1786086081032404 us; error 48 us; skew 500 ppm
I20260807 07:01:21.032804 22229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:01:21 dist-test-slave-ncp9 krb5kdc[21855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086081, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:01:21.036027 22229 init.cc:377] Logged in from keytab as kudu/127.21.83.2@KRBTEST.COM (short username kudu)
I20260807 07:01:21.037606 22229 webserver.cc:460] Webserver started at http://127.21.83.2:46465/ using document root <none> and password file <none>
I20260807 07:01:21.038272 22229 fs_manager.cc:359] Metadata directory not provided
I20260807 07:01:21.038401 22229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:01:21.038725 22229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:01:21.041832 22229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/ts-1/data/instance:
uuid: "162b0091fd704f718c7a478c8a56cd9a"
format_stamp: "Formatted at 2026-08-07 07:01:21 on dist-test-slave-ncp9"
server_key: "276f8e098017751c21795bf55320ac44"
server_key_iv: "9338839dc60a7299320ffd4f2a3d2dda"
server_key_version: "encryptionkey@0"
I20260807 07:01:21.042636 22229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/ts-1/wal/instance:
uuid: "162b0091fd704f718c7a478c8a56cd9a"
format_stamp: "Formatted at 2026-08-07 07:01:21 on dist-test-slave-ncp9"
server_key: "276f8e098017751c21795bf55320ac44"
server_key_iv: "9338839dc60a7299320ffd4f2a3d2dda"
server_key_version: "encryptionkey@0"
I20260807 07:01:21.048101 22229 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 07:01:21.051625 22253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:21.052840 22229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 07:01:21.053050 22229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/ts-1/wal
uuid: "162b0091fd704f718c7a478c8a56cd9a"
format_stamp: "Formatted at 2026-08-07 07:01:21 on dist-test-slave-ncp9"
server_key: "276f8e098017751c21795bf55320ac44"
server_key_iv: "9338839dc60a7299320ffd4f2a3d2dda"
server_key_version: "encryptionkey@0"
I20260807 07:01:21.053234 22229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:01:21.077113 22229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:01:21.080628 22229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:01:21.080880 22229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:01:21.081832 22229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:01:21.083076 22229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:01:21.083153 22229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:21.083271 22229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:01:21.083335 22229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:21.118184 22229 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.2:42585
I20260807 07:01:21.118217 22434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.2:42585 every 8 connection(s)
I20260807 07:01:21.119535 22229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
I20260807 07:01:21.121443 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 22229
I20260807 07:01:21.121649 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/ts-1/wal/instance
I20260807 07:01:21.122123 21836 external_mini_cluster.cc:1428] Setting key 0d45a423aa3d5f360b5371df790a866e
Aug 07 07:01:21 dist-test-slave-ncp9 krb5kdc[21855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086081, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.2@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:01:21.139690 22439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:21.122378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:60135 (local address 127.21.83.62:43965)
0807 07:01:21.122833 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:21.122839 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:21.124006 (+ 1167us) server_negotiation.cc:407] Connection header received
0807 07:01:21.126228 (+ 2222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:21.126232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:21.126294 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:21.126387 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:21.127829 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:21.128514 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:21.130042 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:21.130466 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:21.130574 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:21.133824 (+ 3250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:21.133828 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:21.133832 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:21.133862 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:21.136789 (+ 2927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:21.137484 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:21.137489 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:21.137492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:21.137645 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:21.138004 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:21.138008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:21.138011 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:21.138355 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:21.138491 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:21.139099 (+ 608us) server_negotiation.cc:299] Negotiation successful
0807 07:01:21.139362 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":217,"threads_started":1}
I20260807 07:01:21.140321 22440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:21.122903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:43965 (local address 127.21.83.2:60135)
0807 07:01:21.123830 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:21.123879 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:21.125860 (+ 1981us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:21.126625 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:21.126635 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:21.127121 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:01:21.127665 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:21.127683 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:21.128666 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:21.128670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:21.129900 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:21.129914 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:21.130681 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:21.130689 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:21.130895 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:21.131508 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:01:21.131530 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:01:21.133612 (+ 2082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:01:21.136939 (+ 3327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:21.136945 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:21.136955 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:21.137350 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:21.137736 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:21.137739 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:21.137741 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:21.137903 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:21.138469 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:01:21.138473 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:01:21.138739 (+ 266us) client_negotiation.cc:769] Sending connection context
0807 07:01:21.138951 (+ 212us) client_negotiation.cc:240] Negotiation successful
0807 07:01:21.139305 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":372,"threads_started":1}
I20260807 07:01:21.141662 22435 heartbeater.cc:347] Connected to a master server at 127.21.83.62:43965
I20260807 07:01:21.141989 22435 heartbeater.cc:464] Registering TS with master...
I20260807 07:01:21.142894 22435 heartbeater.cc:511] Master 127.21.83.62:43965 requested a full tablet report, sending...
I20260807 07:01:21.144480 21924 ts_manager.cc:194] Registered new tserver with Master: 162b0091fd704f718c7a478c8a56cd9a (127.21.83.2:42585)
I20260807 07:01:21.145618 21924 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.2@KRBTEST.COM'} at 127.21.83.2:60135
WARNING: no policy specified for kudu/127.21.83.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab.
Entry for principal kudu/127.21.83.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab.
I20260807 07:01:21.175774 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.3:0
--local_ip_for_outbound_sockets=127.21.83.3
--webserver_interface=127.21.83.3
--webserver_port=0
--tserver_master_addrs=127.21.83.62:43965
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab
--principal=kudu/127.21.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:38087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:01:21.576617 22444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:01:21.577092 22444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:01:21.577257 22444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:01:21.584865 22444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:01:21.585014 22444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:01:21.585214 22444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.3
I20260807 07:01:21.595585 22444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:38087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab
--principal=kudu/127.21.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.83.3
--webserver_port=0
--tserver_master_addrs=127.21.83.62:43965
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.3
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:01:21.596983 22444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:01:21.598397 22444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:01:21.612056 22451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:21.612097 22452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:21.612124 22456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:01:21.612885 22444 server_base.cc:1034] running on GCE node
I20260807 07:01:21.613559 22444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:01:21.614553 22444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:01:21.615772 22444 hybrid_clock.cc:648] HybridClock initialized: now 1786086081615715 us; error 49 us; skew 500 ppm
I20260807 07:01:21.616102 22444 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:01:21 dist-test-slave-ncp9 krb5kdc[21855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086081, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:01:21.619323 22444 init.cc:377] Logged in from keytab as kudu/127.21.83.3@KRBTEST.COM (short username kudu)
I20260807 07:01:21.620800 22444 webserver.cc:460] Webserver started at http://127.21.83.3:46695/ using document root <none> and password file <none>
I20260807 07:01:21.621591 22444 fs_manager.cc:359] Metadata directory not provided
I20260807 07:01:21.621716 22444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:01:21.622030 22444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:01:21.625180 22444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/ts-2/data/instance:
uuid: "af37c8586148499a8a8c24e4298ed632"
format_stamp: "Formatted at 2026-08-07 07:01:21 on dist-test-slave-ncp9"
server_key: "b23105373de9c5a7939569b6d98b4a47"
server_key_iv: "fcc3e0d703d48d136ef5c1b42b6d167c"
server_key_version: "encryptionkey@0"
I20260807 07:01:21.626068 22444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/ts-2/wal/instance:
uuid: "af37c8586148499a8a8c24e4298ed632"
format_stamp: "Formatted at 2026-08-07 07:01:21 on dist-test-slave-ncp9"
server_key: "b23105373de9c5a7939569b6d98b4a47"
server_key_iv: "fcc3e0d703d48d136ef5c1b42b6d167c"
server_key_version: "encryptionkey@0"
I20260807 07:01:21.628458 22219 heartbeater.cc:503] Master 127.21.83.62:43965 was elected leader, sending a full tablet report...
I20260807 07:01:21.632150 22444 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260807 07:01:21.636303 22465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:21.637842 22444 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 07:01:21.638038 22444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/ts-2/wal
uuid: "af37c8586148499a8a8c24e4298ed632"
format_stamp: "Formatted at 2026-08-07 07:01:21 on dist-test-slave-ncp9"
server_key: "b23105373de9c5a7939569b6d98b4a47"
server_key_iv: "fcc3e0d703d48d136ef5c1b42b6d167c"
server_key_version: "encryptionkey@0"
I20260807 07:01:21.638259 22444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:01:21.658782 22444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:01:21.663133 22444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:01:21.663463 22444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:01:21.664520 22444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:01:21.666009 22444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:01:21.666105 22444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:21.666235 22444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:01:21.666289 22444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:21.703092 22444 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.3:43581
I20260807 07:01:21.703135 22642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.3:43581 every 8 connection(s)
I20260807 07:01:21.704510 22444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
I20260807 07:01:21.708526 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 22444
I20260807 07:01:21.708726 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786086078080656-21836-0/minicluster-data/ts-2/wal/instance
I20260807 07:01:21.709314 21836 external_mini_cluster.cc:1428] Setting key 981b2f1d17c3ef8db9bf439cf3a1606d
Aug 07 07:01:21 dist-test-slave-ncp9 krb5kdc[21855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086081, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.3@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:01:21.724684 22646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:21.707119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:56403 (local address 127.21.83.62:43965)
0807 07:01:21.707583 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:21.707587 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:21.708553 (+ 966us) server_negotiation.cc:407] Connection header received
0807 07:01:21.711075 (+ 2522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:21.711081 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:21.711185 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:21.711272 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:21.713036 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:21.713937 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:21.715484 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:21.715896 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:21.716005 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:21.719242 (+ 3237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:21.719247 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:21.719250 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:21.719284 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:21.721616 (+ 2332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:21.722319 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:21.722323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:21.722325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:21.722445 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:21.722866 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:21.722870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:21.722872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:21.723137 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:21.723293 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:21.724125 (+ 832us) server_negotiation.cc:299] Negotiation successful
0807 07:01:21.724337 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":238,"threads_started":1}
I20260807 07:01:21.725747 22647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:21.707557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:43965 (local address 127.21.83.3:56403)
0807 07:01:21.708362 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:21.708411 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:21.710755 (+ 2344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:21.711555 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:21.711567 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:21.712222 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:01:21.712780 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:21.712800 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:21.714074 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:21.714078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:21.715332 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:21.715345 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:21.716086 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:21.716091 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:21.716277 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:21.716858 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:01:21.716885 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:01:21.719024 (+ 2139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:01:21.721751 (+ 2727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:21.721760 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:21.721776 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:21.722179 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:21.722548 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:21.722553 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:21.722557 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:21.722753 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:21.723273 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:01:21.723282 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:01:21.723602 (+ 320us) client_negotiation.cc:769] Sending connection context
0807 07:01:21.723981 (+ 379us) client_negotiation.cc:240] Negotiation successful
0807 07:01:21.724452 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":296,"threads_started":1}
I20260807 07:01:21.726994 22643 heartbeater.cc:347] Connected to a master server at 127.21.83.62:43965
I20260807 07:01:21.727275 22643 heartbeater.cc:464] Registering TS with master...
I20260807 07:01:21.728017 22643 heartbeater.cc:511] Master 127.21.83.62:43965 requested a full tablet report, sending...
I20260807 07:01:21.729480 21924 ts_manager.cc:194] Registered new tserver with Master: af37c8586148499a8a8c24e4298ed632 (127.21.83.3:43581)
I20260807 07:01:21.730268 21924 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.3@KRBTEST.COM'} at 127.21.83.3:56403
I20260807 07:01:21.738195 21836 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 07:01:21 dist-test-slave-ncp9 krb5kdc[21855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086081, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 07:01:21 dist-test-slave-ncp9 krb5kdc[21855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086081, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.83.1@KRBTEST.COM
I20260807 07:01:21.786361 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:21.771352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46402 (local address 127.21.83.1:35813)
0807 07:01:21.772048 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:21.772058 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:21.772091 (+ 33us) server_negotiation.cc:407] Connection header received
0807 07:01:21.772160 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:21.772167 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:21.772380 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:21.772520 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:21.774005 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:21.775008 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:21.776259 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:21.776812 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:21.776950 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:21.780754 (+ 3804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:21.780765 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:21.780779 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:21.780812 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:21.783558 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:21.784213 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:21.784220 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:21.784223 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:21.784349 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:21.784738 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:21.784742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:21.784744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:21.785184 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:21.785470 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:21.785759 (+ 289us) server_negotiation.cc:299] Negotiation successful
0807 07:01:21.786020 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":335,"threads_started":1}
W20260807 07:01:21.787707 22089 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46402
Aug 07 07:01:21 dist-test-slave-ncp9 krb5kdc[21855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 07:01:21 dist-test-slave-ncp9 krb5kdc[21855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.1@KRBTEST.COM
I20260807 07:01:21.832679 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:21.819397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46408 (local address 127.21.83.1:35813)
0807 07:01:21.819595 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:21.819601 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:21.819873 (+ 272us) server_negotiation.cc:407] Connection header received
0807 07:01:21.820047 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:21.820051 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:21.820116 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:21.820200 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:21.821566 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:21.822269 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:21.823458 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:21.823887 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:21.823996 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:21.827611 (+ 3615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:21.827616 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:21.827619 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:21.827651 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:21.829978 (+ 2327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:21.830558 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:21.830562 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:21.830564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:21.830699 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:21.831231 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:21.831236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:21.831238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:21.831526 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:21.831776 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:21.832104 (+ 328us) server_negotiation.cc:299] Negotiation successful
0807 07:01:21.832345 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 07:01:21.834372 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 22019
I20260807 07:01:21.847872 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:21.835397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46416 (local address 127.21.83.1:35813)
0807 07:01:21.835583 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:21.835589 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:21.835835 (+ 246us) server_negotiation.cc:407] Connection header received
0807 07:01:21.835992 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:21.835998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:21.836065 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:21.836144 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:21.837406 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:21.838099 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:21.839459 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:21.839971 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:21.840087 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:21.842450 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:21.842455 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:21.842458 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:21.842490 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:21.845260 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:21.846016 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:21.846022 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:21.846024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:21.846203 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:21.846509 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:21.846512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:21.846514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:21.846830 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:21.847040 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:21.847271 (+ 231us) server_negotiation.cc:299] Negotiation successful
0807 07:01:21.847471 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260807 07:01:21.848846 22211 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:01:21.936195 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 22019
I20260807 07:01:21.959023 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 22229
Aug 07 07:01:21 dist-test-slave-ncp9 krb5kdc[21855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.2@KRBTEST.COM
I20260807 07:01:21.973445 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:21.959873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40718 (local address 127.21.83.2:42585)
0807 07:01:21.960433 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:21.960440 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:21.960465 (+ 25us) server_negotiation.cc:407] Connection header received
0807 07:01:21.960528 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:21.960534 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:21.960742 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:21.960866 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:21.962030 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:21.963055 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:21.964212 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:21.964736 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:21.964829 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:21.968644 (+ 3815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:21.968656 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:21.968669 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:21.968710 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:21.971177 (+ 2467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:21.971738 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:21.971744 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:21.971747 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:21.971859 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:21.972163 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:21.972166 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:21.972168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:21.972483 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:21.972720 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:21.972935 (+ 215us) server_negotiation.cc:299] Negotiation successful
0807 07:01:21.973162 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":269,"threads_started":1}
I20260807 07:01:21.974653 22427 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:01:22.056633 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 22229
I20260807 07:01:22.077428 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 22444
Aug 07 07:01:22 dist-test-slave-ncp9 krb5kdc[21855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.3@KRBTEST.COM
I20260807 07:01:22.092883 22667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:22.078385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47028 (local address 127.21.83.3:43581)
0807 07:01:22.078996 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:22.079003 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:22.079033 (+ 30us) server_negotiation.cc:407] Connection header received
0807 07:01:22.079098 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:22.079104 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:22.079318 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:22.079451 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:22.080601 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:22.081662 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:22.082875 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:22.083407 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:22.083500 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:22.087307 (+ 3807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:22.087320 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:22.087334 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:22.087372 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:22.090125 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:22.090748 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:22.090754 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:22.090756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:22.090887 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:22.091310 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:22.091314 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:22.091316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:22.091682 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:22.091933 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:22.092300 (+ 367us) server_negotiation.cc:299] Negotiation successful
0807 07:01:22.092550 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":303,"threads_started":1}
I20260807 07:01:22.094143 22636 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:01:22.175460 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 22444
I20260807 07:01:22.195397 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 21872
I20260807 07:01:22.196494 21969 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:01:22.302605 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 21872
2026-08-07T07:01:22Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4192 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 07:01:22 dist-test-slave-ncp9 krb5kdc[22674](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 07:01:22 dist-test-slave-ncp9 krb5kdc[22674](info): set up 2 sockets
Aug 07 07:01:22 dist-test-slave-ncp9 krb5kdc[22674](info): commencing operation
krb5kdc: starting...
W20260807 07:01:23.487262 21836 mini_kdc.cc:121] Time spent starting KDC: real 1.137s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Aug 07 07:01:23 dist-test-slave-ncp9 krb5kdc[22674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T07:01:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:01:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
I20260807 07:01:23.700507 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:38191
--webserver_interface=127.21.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:35965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:38191
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:01:24.047442 22697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:01:24.047775 22697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:01:24.047900 22697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:01:24.055629 22697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:01:24.055779 22697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:01:24.055836 22697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:01:24.055917 22697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:01:24.055974 22697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:01:24.066439 22697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:35965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:38191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:38191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:01:24.067832 22697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:01:24.069217 22697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:01:24.080605 22704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:24.080718 22703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:24.080652 22706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:01:24.081414 22697 server_base.cc:1034] running on GCE node
I20260807 07:01:24.082131 22697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:01:24.083437 22697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:01:24.084622 22697 hybrid_clock.cc:648] HybridClock initialized: now 1786086084084564 us; error 46 us; skew 500 ppm
I20260807 07:01:24.084977 22697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:01:24 dist-test-slave-ncp9 krb5kdc[22674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086084, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:01:24.087903 22697 init.cc:377] Logged in from keytab as kudu/127.21.83.62@KRBTEST.COM (short username kudu)
I20260807 07:01:24.089289 22697 webserver.cc:460] Webserver started at http://127.21.83.62:33103/ using document root <none> and password file <none>
I20260807 07:01:24.089916 22697 fs_manager.cc:359] Metadata directory not provided
I20260807 07:01:24.090037 22697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:01:24.090320 22697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:01:24.093163 22697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "61353e07b43e46b4a7a05b2ba44852f2"
format_stamp: "Formatted at 2026-08-07 07:01:24 on dist-test-slave-ncp9"
server_key: "39e3b7ae2e1018921d68a51469cb18f3"
server_key_iv: "e9a4dab144f611ba74da003f7af37df5"
server_key_version: "encryptionkey@0"
I20260807 07:01:24.093853 22697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "61353e07b43e46b4a7a05b2ba44852f2"
format_stamp: "Formatted at 2026-08-07 07:01:24 on dist-test-slave-ncp9"
server_key: "39e3b7ae2e1018921d68a51469cb18f3"
server_key_iv: "e9a4dab144f611ba74da003f7af37df5"
server_key_version: "encryptionkey@0"
I20260807 07:01:24.098744 22697 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 07:01:24.102049 22717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:24.103271 22697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 07:01:24.103422 22697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "61353e07b43e46b4a7a05b2ba44852f2"
format_stamp: "Formatted at 2026-08-07 07:01:24 on dist-test-slave-ncp9"
server_key: "39e3b7ae2e1018921d68a51469cb18f3"
server_key_iv: "e9a4dab144f611ba74da003f7af37df5"
server_key_version: "encryptionkey@0"
I20260807 07:01:24.103600 22697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:01:24.120640 22697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:01:24.125200 22697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:01:24.125473 22697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:01:24.142410 22697 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.62:38191
I20260807 07:01:24.142457 22794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.62:38191 every 8 connection(s)
I20260807 07:01:24.143685 22697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:01:24.147132 22796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:01:24.152385 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 22697
I20260807 07:01:24.152622 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:01:24.153492 21836 external_mini_cluster.cc:1428] Setting key 13c99d84043a32b837428f3e43e132d9
I20260807 07:01:24.158033 22796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61353e07b43e46b4a7a05b2ba44852f2: Bootstrap starting.
I20260807 07:01:24.161389 22796 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61353e07b43e46b4a7a05b2ba44852f2: Neither blocks nor log segments found. Creating new log.
I20260807 07:01:24.162506 22796 log.cc:824] T 00000000000000000000000000000000 P 61353e07b43e46b4a7a05b2ba44852f2: Log is configured to *not* fsync() on all Append() calls
I20260807 07:01:24.165273 22796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61353e07b43e46b4a7a05b2ba44852f2: No bootstrap required, opened a new log
Aug 07 07:01:24 dist-test-slave-ncp9 krb5kdc[22674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:01:24.173482 22796 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61353e07b43e46b4a7a05b2ba44852f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61353e07b43e46b4a7a05b2ba44852f2" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 38191 } }
I20260807 07:01:24.173902 22796 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61353e07b43e46b4a7a05b2ba44852f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:01:24.174001 22796 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61353e07b43e46b4a7a05b2ba44852f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61353e07b43e46b4a7a05b2ba44852f2, State: Initialized, Role: FOLLOWER
I20260807 07:01:24.174618 22796 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61353e07b43e46b4a7a05b2ba44852f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61353e07b43e46b4a7a05b2ba44852f2" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 38191 } }
I20260807 07:01:24.174868 22796 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61353e07b43e46b4a7a05b2ba44852f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:01:24.175004 22796 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61353e07b43e46b4a7a05b2ba44852f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:01:24.175195 22796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61353e07b43e46b4a7a05b2ba44852f2 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:01:24.176800 22800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:24.155450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49232 (local address 127.21.83.62:38191)
0807 07:01:24.156223 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:24.156234 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:24.156273 (+ 39us) server_negotiation.cc:407] Connection header received
0807 07:01:24.158012 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:24.158038 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:24.158401 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:24.158703 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:24.160466 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:24.161795 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:24.163411 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:24.164140 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:24.164264 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:24.168706 (+ 4442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:24.168715 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:24.168731 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:24.168782 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:24.172343 (+ 3561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:24.173251 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:24.173259 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:24.173266 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:24.173448 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:24.173837 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:24.173841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:24.173843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:24.174480 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:24.174813 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:24.175294 (+ 481us) server_negotiation.cc:299] Negotiation successful
0807 07:01:24.175737 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":361,"threads_started":1}
I20260807 07:01:24.177724 22796 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61353e07b43e46b4a7a05b2ba44852f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61353e07b43e46b4a7a05b2ba44852f2" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 38191 } }
I20260807 07:01:24.178246 22796 leader_election.cc:302] T 00000000000000000000000000000000 P 61353e07b43e46b4a7a05b2ba44852f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61353e07b43e46b4a7a05b2ba44852f2; no voters:
I20260807 07:01:24.178898 22796 leader_election.cc:288] T 00000000000000000000000000000000 P 61353e07b43e46b4a7a05b2ba44852f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 07:01:24.179021 22802 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61353e07b43e46b4a7a05b2ba44852f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:01:24.179935 22802 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61353e07b43e46b4a7a05b2ba44852f2 [term 1 LEADER]: Becoming Leader. State: Replica: 61353e07b43e46b4a7a05b2ba44852f2, State: Running, Role: LEADER
I20260807 07:01:24.180573 22802 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61353e07b43e46b4a7a05b2ba44852f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61353e07b43e46b4a7a05b2ba44852f2" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 38191 } }
I20260807 07:01:24.182034 22796 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61353e07b43e46b4a7a05b2ba44852f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:01:24.183527 22803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61353e07b43e46b4a7a05b2ba44852f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61353e07b43e46b4a7a05b2ba44852f2. Latest consensus state: current_term: 1 leader_uuid: "61353e07b43e46b4a7a05b2ba44852f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61353e07b43e46b4a7a05b2ba44852f2" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 38191 } } }
I20260807 07:01:24.183938 22803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61353e07b43e46b4a7a05b2ba44852f2 [sys.catalog]: This master's current role is: LEADER
I20260807 07:01:24.184201 22804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61353e07b43e46b4a7a05b2ba44852f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61353e07b43e46b4a7a05b2ba44852f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61353e07b43e46b4a7a05b2ba44852f2" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 38191 } } }
I20260807 07:01:24.184612 22804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61353e07b43e46b4a7a05b2ba44852f2 [sys.catalog]: This master's current role is: LEADER
I20260807 07:01:24.189750 22809 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:01:24.194849 22809 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:01:24.203326 22809 catalog_manager.cc:1347] Generated new cluster ID: 784e579fa577452c9b941c35d9ae3d9b
I20260807 07:01:24.203414 22809 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:01:24.214004 22809 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:01:24.215198 22809 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:01:24.221010 22809 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61353e07b43e46b4a7a05b2ba44852f2: Generated new TSK 0
I20260807 07:01:24.221746 22809 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.83.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab.
Entry for principal kudu/127.21.83.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab.
I20260807 07:01:24.286406 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.1:0
--local_ip_for_outbound_sockets=127.21.83.1
--webserver_interface=127.21.83.1
--webserver_port=0
--tserver_master_addrs=127.21.83.62:38191
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab
--principal=kudu/127.21.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:35965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:01:24.633423 22833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:01:24.633711 22833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:01:24.633860 22833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:01:24.641052 22833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:01:24.641186 22833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:01:24.641359 22833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.1
I20260807 07:01:24.651252 22833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:35965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab
--principal=kudu/127.21.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.83.1
--webserver_port=0
--tserver_master_addrs=127.21.83.62:38191
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.1
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:01:24.652556 22833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:01:24.653873 22833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:01:24.666112 22843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:24.666074 22844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:24.666033 22846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:01:24.667078 22833 server_base.cc:1034] running on GCE node
I20260807 07:01:24.667634 22833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:01:24.668494 22833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:01:24.669678 22833 hybrid_clock.cc:648] HybridClock initialized: now 1786086084669630 us; error 40 us; skew 500 ppm
I20260807 07:01:24.669935 22833 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:01:24 dist-test-slave-ncp9 krb5kdc[22674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086084, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:01:24.672776 22833 init.cc:377] Logged in from keytab as kudu/127.21.83.1@KRBTEST.COM (short username kudu)
I20260807 07:01:24.674058 22833 webserver.cc:460] Webserver started at http://127.21.83.1:45385/ using document root <none> and password file <none>
I20260807 07:01:24.674701 22833 fs_manager.cc:359] Metadata directory not provided
I20260807 07:01:24.674814 22833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:01:24.675097 22833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:01:24.678004 22833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/ts-0/data/instance:
uuid: "3ce3d28be1cd440f85cb208a81b04393"
format_stamp: "Formatted at 2026-08-07 07:01:24 on dist-test-slave-ncp9"
server_key: "98dc01298846a037ff750c626eb0175f"
server_key_iv: "7502401e2faefd7f34323ef5bc6fcdbd"
server_key_version: "encryptionkey@0"
I20260807 07:01:24.678750 22833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/ts-0/wal/instance:
uuid: "3ce3d28be1cd440f85cb208a81b04393"
format_stamp: "Formatted at 2026-08-07 07:01:24 on dist-test-slave-ncp9"
server_key: "98dc01298846a037ff750c626eb0175f"
server_key_iv: "7502401e2faefd7f34323ef5bc6fcdbd"
server_key_version: "encryptionkey@0"
I20260807 07:01:24.683507 22833 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 07:01:24.686762 22855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:24.688094 22833 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 07:01:24.688241 22833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/ts-0/wal
uuid: "3ce3d28be1cd440f85cb208a81b04393"
format_stamp: "Formatted at 2026-08-07 07:01:24 on dist-test-slave-ncp9"
server_key: "98dc01298846a037ff750c626eb0175f"
server_key_iv: "7502401e2faefd7f34323ef5bc6fcdbd"
server_key_version: "encryptionkey@0"
I20260807 07:01:24.688418 22833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:01:24.706768 22833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:01:24.710736 22833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:01:24.710999 22833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:01:24.711897 22833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:01:24.713238 22833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:01:24.713316 22833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:24.713438 22833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:01:24.713491 22833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:24.743216 22833 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.1:46291
I20260807 07:01:24.743254 23022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.1:46291 every 8 connection(s)
I20260807 07:01:24.744552 22833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
I20260807 07:01:24.749348 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 22833
I20260807 07:01:24.749562 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/ts-0/wal/instance
I20260807 07:01:24.750195 21836 external_mini_cluster.cc:1428] Setting key b2f62b03a26c8a1dd55f2648449a3d75
Aug 07 07:01:24 dist-test-slave-ncp9 krb5kdc[22674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086084, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.1@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:01:24.765662 23026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:24.747115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:41233 (local address 127.21.83.62:38191)
0807 07:01:24.747552 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:24.747556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:24.748414 (+ 858us) server_negotiation.cc:407] Connection header received
0807 07:01:24.750525 (+ 2111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:24.750531 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:24.750614 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:24.750716 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:24.752495 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:24.753319 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:24.754939 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:24.755361 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:24.755483 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:24.759316 (+ 3833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:24.759322 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:24.759325 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:24.759364 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:24.761928 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:24.762974 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:24.762979 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:24.762982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:24.763138 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:24.763739 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:24.763744 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:24.763746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:24.764062 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:24.764182 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:24.765136 (+ 954us) server_negotiation.cc:299] Negotiation successful
0807 07:01:24.765354 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":224,"threads_started":1}
I20260807 07:01:24.767314 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:24.747540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38191 (local address 127.21.83.1:41233)
0807 07:01:24.748229 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:24.748287 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:24.750212 (+ 1925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:24.750999 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:24.751011 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:24.751636 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:01:24.752321 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:24.752342 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:24.753490 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:24.753493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:24.754784 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:24.754799 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:24.755628 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:24.755637 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:24.755874 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:24.756600 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:01:24.756632 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:01:24.759047 (+ 2415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:01:24.762115 (+ 3068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:24.762125 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:24.762145 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:24.762804 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:24.763274 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:24.763279 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:24.763283 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:24.763614 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:24.764197 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:01:24.764206 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:01:24.764672 (+ 466us) client_negotiation.cc:769] Sending connection context
0807 07:01:24.765046 (+ 374us) client_negotiation.cc:240] Negotiation successful
0807 07:01:24.765704 (+ 658us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":275,"threads_started":1}
I20260807 07:01:24.769150 23023 heartbeater.cc:347] Connected to a master server at 127.21.83.62:38191
I20260807 07:01:24.769472 23023 heartbeater.cc:464] Registering TS with master...
I20260807 07:01:24.770254 23023 heartbeater.cc:511] Master 127.21.83.62:38191 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.21.83.2@KRBTEST.COM; defaulting to no policy
I20260807 07:01:24.772821 22746 ts_manager.cc:194] Registered new tserver with Master: 3ce3d28be1cd440f85cb208a81b04393 (127.21.83.1:46291)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.2@KRBTEST.COM" created.
I20260807 07:01:24.774470 22746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.1@KRBTEST.COM'} at 127.21.83.1:41233
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab.
Entry for principal kudu/127.21.83.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab.
I20260807 07:01:24.798331 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.2:0
--local_ip_for_outbound_sockets=127.21.83.2
--webserver_interface=127.21.83.2
--webserver_port=0
--tserver_master_addrs=127.21.83.62:38191
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab
--principal=kudu/127.21.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:35965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:01:25.129863 23034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:01:25.130120 23034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:01:25.130216 23034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:01:25.137091 23034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:01:25.137183 23034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:01:25.137305 23034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.2
I20260807 07:01:25.149410 23034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:35965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab
--principal=kudu/127.21.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.83.2
--webserver_port=0
--tserver_master_addrs=127.21.83.62:38191
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.2
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:01:25.150717 23034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:01:25.152215 23034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:01:25.165580 23040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:25.165513 23044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:25.165679 23048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:01:25.166352 23034 server_base.cc:1034] running on GCE node
I20260807 07:01:25.166872 23034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:01:25.167670 23034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:01:25.168853 23034 hybrid_clock.cc:648] HybridClock initialized: now 1786086085168807 us; error 40 us; skew 500 ppm
I20260807 07:01:25.169150 23034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:01:25 dist-test-slave-ncp9 krb5kdc[22674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086085, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:01:25.176755 23034 init.cc:377] Logged in from keytab as kudu/127.21.83.2@KRBTEST.COM (short username kudu)
I20260807 07:01:25.178107 23034 webserver.cc:460] Webserver started at http://127.21.83.2:34825/ using document root <none> and password file <none>
I20260807 07:01:25.178716 23034 fs_manager.cc:359] Metadata directory not provided
I20260807 07:01:25.178829 23034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:01:25.179105 23034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:01:25.181821 23034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/ts-1/data/instance:
uuid: "b49b2228389944659526ae92e017a572"
format_stamp: "Formatted at 2026-08-07 07:01:25 on dist-test-slave-ncp9"
server_key: "30cc2961b059bb170f530ee637924979"
server_key_iv: "851fb0af2decc523f7b9d040597c6064"
server_key_version: "encryptionkey@0"
I20260807 07:01:25.182591 23034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/ts-1/wal/instance:
uuid: "b49b2228389944659526ae92e017a572"
format_stamp: "Formatted at 2026-08-07 07:01:25 on dist-test-slave-ncp9"
server_key: "30cc2961b059bb170f530ee637924979"
server_key_iv: "851fb0af2decc523f7b9d040597c6064"
server_key_version: "encryptionkey@0"
I20260807 07:01:25.187310 23034 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 07:01:25.190443 23058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:25.191625 23034 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260807 07:01:25.191771 23034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/ts-1/wal
uuid: "b49b2228389944659526ae92e017a572"
format_stamp: "Formatted at 2026-08-07 07:01:25 on dist-test-slave-ncp9"
server_key: "30cc2961b059bb170f530ee637924979"
server_key_iv: "851fb0af2decc523f7b9d040597c6064"
server_key_version: "encryptionkey@0"
I20260807 07:01:25.191952 23034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:01:25.209331 23034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:01:25.213718 23034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:01:25.213989 23034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:01:25.215008 23034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:01:25.216441 23034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:01:25.216528 23034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:25.216629 23034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:01:25.216672 23034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:25.248382 23034 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.2:42537
I20260807 07:01:25.248494 23229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.2:42537 every 8 connection(s)
I20260807 07:01:25.249693 23034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
I20260807 07:01:25.258399 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 23034
I20260807 07:01:25.258699 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/ts-1/wal/instance
I20260807 07:01:25.259276 21836 external_mini_cluster.cc:1428] Setting key 1ae6034b9a73913d257924cc1db86353
Aug 07 07:01:25 dist-test-slave-ncp9 krb5kdc[22674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086085, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.2@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:01:25.270713 23026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:25.252196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:55293 (local address 127.21.83.62:38191)
0807 07:01:25.252419 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:25.252425 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:25.253557 (+ 1132us) server_negotiation.cc:407] Connection header received
0807 07:01:25.255462 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:25.255474 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:25.255547 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:25.255636 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:25.257194 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:25.257898 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:25.260061 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:25.260644 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:25.260783 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:25.264757 (+ 3974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:25.264762 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:25.264765 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:25.264802 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:25.267319 (+ 2517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:25.268228 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:25.268233 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:25.268235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:25.268368 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:25.268856 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:25.268860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:25.268862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:25.269210 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:25.269356 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:25.270118 (+ 762us) server_negotiation.cc:299] Negotiation successful
0807 07:01:25.270372 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 07:01:25.271786 23232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:25.252660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38191 (local address 127.21.83.2:55293)
0807 07:01:25.253387 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:25.253429 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:25.255197 (+ 1768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:25.255873 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:25.255881 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:25.256402 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:01:25.257025 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:25.257045 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:25.258035 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:25.258040 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:25.259917 (+ 1877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:25.259930 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:25.260862 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:25.260867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:25.261118 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:25.261801 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:01:25.261831 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:01:25.264506 (+ 2675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:01:25.267470 (+ 2964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:25.267487 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:25.267504 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:25.268103 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:25.268494 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:25.268499 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:25.268502 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:25.268725 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:25.269340 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:01:25.269349 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:01:25.269721 (+ 372us) client_negotiation.cc:769] Sending connection context
0807 07:01:25.269974 (+ 253us) client_negotiation.cc:240] Negotiation successful
0807 07:01:25.270543 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":295,"threads_started":1}
I20260807 07:01:25.273479 23230 heartbeater.cc:347] Connected to a master server at 127.21.83.62:38191
I20260807 07:01:25.273809 23230 heartbeater.cc:464] Registering TS with master...
I20260807 07:01:25.274873 23230 heartbeater.cc:511] Master 127.21.83.62:38191 requested a full tablet report, sending...
I20260807 07:01:25.276515 22746 ts_manager.cc:194] Registered new tserver with Master: b49b2228389944659526ae92e017a572 (127.21.83.2:42537)
I20260807 07:01:25.277734 22746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.2@KRBTEST.COM'} at 127.21.83.2:55293
WARNING: no policy specified for kudu/127.21.83.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab.
Entry for principal kudu/127.21.83.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab.
I20260807 07:01:25.306001 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.3:0
--local_ip_for_outbound_sockets=127.21.83.3
--webserver_interface=127.21.83.3
--webserver_port=0
--tserver_master_addrs=127.21.83.62:38191
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab
--principal=kudu/127.21.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:35965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:01:25.636786 23239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:01:25.637146 23239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:01:25.637290 23239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:01:25.646099 23239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:01:25.646260 23239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:01:25.646428 23239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.3
I20260807 07:01:25.656847 23239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:35965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab
--principal=kudu/127.21.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.83.3
--webserver_port=0
--tserver_master_addrs=127.21.83.62:38191
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.3
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:01:25.658308 23239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:01:25.659826 23239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:01:25.673938 23252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:25.674068 23253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:25.673967 23258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:01:25.674777 23239 server_base.cc:1034] running on GCE node
I20260807 07:01:25.675454 23239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:01:25.676429 23239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:01:25.677656 23239 hybrid_clock.cc:648] HybridClock initialized: now 1786086085677600 us; error 42 us; skew 500 ppm
I20260807 07:01:25.677989 23239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:01:25 dist-test-slave-ncp9 krb5kdc[22674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086085, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:01:25.681085 23239 init.cc:377] Logged in from keytab as kudu/127.21.83.3@KRBTEST.COM (short username kudu)
I20260807 07:01:25.682560 23239 webserver.cc:460] Webserver started at http://127.21.83.3:34655/ using document root <none> and password file <none>
I20260807 07:01:25.683257 23239 fs_manager.cc:359] Metadata directory not provided
I20260807 07:01:25.683379 23239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:01:25.683678 23239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:01:25.686586 23239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/ts-2/data/instance:
uuid: "f6275135901a45b492bd4643e41ac801"
format_stamp: "Formatted at 2026-08-07 07:01:25 on dist-test-slave-ncp9"
server_key: "d06641a4f1d64ad3457101de695e0f69"
server_key_iv: "f7b3bca5df9d9ef5d9ca3afe0782156e"
server_key_version: "encryptionkey@0"
I20260807 07:01:25.687348 23239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/ts-2/wal/instance:
uuid: "f6275135901a45b492bd4643e41ac801"
format_stamp: "Formatted at 2026-08-07 07:01:25 on dist-test-slave-ncp9"
server_key: "d06641a4f1d64ad3457101de695e0f69"
server_key_iv: "f7b3bca5df9d9ef5d9ca3afe0782156e"
server_key_version: "encryptionkey@0"
I20260807 07:01:25.692549 23239 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 07:01:25.696429 23267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:25.697870 23239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 07:01:25.698038 23239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/ts-2/wal
uuid: "f6275135901a45b492bd4643e41ac801"
format_stamp: "Formatted at 2026-08-07 07:01:25 on dist-test-slave-ncp9"
server_key: "d06641a4f1d64ad3457101de695e0f69"
server_key_iv: "f7b3bca5df9d9ef5d9ca3afe0782156e"
server_key_version: "encryptionkey@0"
I20260807 07:01:25.698241 23239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:01:25.714685 23239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:01:25.719123 23239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:01:25.719417 23239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:01:25.720455 23239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:01:25.721951 23239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:01:25.722031 23239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:25.722137 23239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:01:25.722209 23239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:25.754868 23239 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.3:37641
I20260807 07:01:25.754879 23426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.3:37641 every 8 connection(s)
I20260807 07:01:25.756227 23239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
I20260807 07:01:25.765223 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 23239
I20260807 07:01:25.765426 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786086078080656-21836-0/minicluster-data/ts-2/wal/instance
I20260807 07:01:25.765841 21836 external_mini_cluster.cc:1428] Setting key fa4c6b8edbfc60f96f5b2bf443742543
Aug 07 07:01:25 dist-test-slave-ncp9 krb5kdc[22674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086085, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.3@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:01:25.775604 23026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:25.758940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:46527 (local address 127.21.83.62:38191)
0807 07:01:25.759147 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:25.759154 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:25.760390 (+ 1236us) server_negotiation.cc:407] Connection header received
0807 07:01:25.762232 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:25.762237 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:25.762335 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:25.762420 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:25.763864 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:25.764987 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:25.766623 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:25.767193 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:25.767323 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:25.770674 (+ 3351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:25.770680 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:25.770683 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:25.770715 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:25.773088 (+ 2373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:25.773746 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:25.773749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:25.773752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:25.773873 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:25.774236 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:25.774239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:25.774241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:25.774497 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:25.774609 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:25.775194 (+ 585us) server_negotiation.cc:299] Negotiation successful
0807 07:01:25.775355 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 07:01:25.776484 23433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:25.759421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38191 (local address 127.21.83.3:46527)
0807 07:01:25.760234 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:25.760272 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:25.761955 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:25.762628 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:25.762635 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:25.763146 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:01:25.763696 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:25.763719 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:25.765132 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:25.765136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:25.766483 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:25.766498 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:25.767393 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:25.767398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:25.767637 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:25.768296 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:01:25.768317 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:01:25.770501 (+ 2184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:01:25.773233 (+ 2732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:25.773240 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:25.773252 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:25.773653 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:25.773954 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:25.773957 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:25.773959 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:25.774149 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:25.774591 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:01:25.774597 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:01:25.774890 (+ 293us) client_negotiation.cc:769] Sending connection context
0807 07:01:25.775107 (+ 217us) client_negotiation.cc:240] Negotiation successful
0807 07:01:25.775476 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":313,"threads_started":1}
I20260807 07:01:25.777598 23023 heartbeater.cc:503] Master 127.21.83.62:38191 was elected leader, sending a full tablet report...
I20260807 07:01:25.777987 23428 heartbeater.cc:347] Connected to a master server at 127.21.83.62:38191
I20260807 07:01:25.778290 23428 heartbeater.cc:464] Registering TS with master...
I20260807 07:01:25.779016 23428 heartbeater.cc:511] Master 127.21.83.62:38191 requested a full tablet report, sending...
I20260807 07:01:25.780500 22746 ts_manager.cc:194] Registered new tserver with Master: f6275135901a45b492bd4643e41ac801 (127.21.83.3:37641)
I20260807 07:01:25.781133 21836 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 07:01:25.781486 22746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.3@KRBTEST.COM'} at 127.21.83.3:46527
I20260807 07:01:25.800256 23026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:25.787479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49240 (local address 127.21.83.62:38191)
0807 07:01:25.787637 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:25.787642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:25.788117 (+ 475us) server_negotiation.cc:407] Connection header received
0807 07:01:25.788285 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:25.788289 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:25.788348 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:25.788427 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:25.789962 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:25.790755 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:25.792039 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:25.792492 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:25.792601 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:25.794837 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:25.794843 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:25.794846 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:25.794876 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:25.797670 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:25.798302 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:25.798306 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:25.798308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:25.798435 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:25.798798 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:25.798801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:25.798803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:25.799138 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:25.799384 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:25.799719 (+ 335us) server_negotiation.cc:299] Negotiation successful
0807 07:01:25.799945 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 07:01:25.807328 22746 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49240:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 07:01:25.814227 22746 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 07:01:25.839707 23449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:25.827205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:46291 (local address 127.0.0.1:40922)
0807 07:01:25.829432 (+ 2227us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:25.829480 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:25.829772 (+ 292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:25.830850 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:25.830859 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:25.830899 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:01:25.831337 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:25.831350 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:25.833545 (+ 2195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:25.833551 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:25.836062 (+ 2511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:25.836080 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:25.838545 (+ 2465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:25.838551 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:25.838814 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:25.838870 (+ 56us) client_negotiation.cc:769] Sending connection context
0807 07:01:25.839014 (+ 144us) client_negotiation.cc:240] Negotiation successful
0807 07:01:25.839247 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1978,"thread_start_us":1811,"threads_started":1}
I20260807 07:01:25.840917 23450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:25.827330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40922 (local address 127.21.83.1:46291)
0807 07:01:25.829031 (+ 1701us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:25.829040 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:25.829593 (+ 553us) server_negotiation.cc:407] Connection header received
0807 07:01:25.830022 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:25.830030 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:25.830305 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:25.830520 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:01:25.831556 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:25.833346 (+ 1790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:25.836290 (+ 2944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:25.837931 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:25.838045 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:25.838328 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:25.840287 (+ 1959us) server_negotiation.cc:299] Negotiation successful
0807 07:01:25.840524 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1497,"thread_start_us":1324,"threads_started":1}
I20260807 07:01:25.843848 23451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:25.827732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.2:42537 (local address 127.0.0.1:39208)
0807 07:01:25.829955 (+ 2223us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:25.829972 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:25.830092 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:25.832345 (+ 2253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:25.832351 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:25.832378 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:01:25.832825 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:25.832836 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:25.836074 (+ 3238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:25.836079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:25.838359 (+ 2280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:25.838379 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:25.842951 (+ 4572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:25.842959 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:25.843178 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:25.843213 (+ 35us) client_negotiation.cc:769] Sending connection context
0807 07:01:25.843312 (+ 99us) client_negotiation.cc:240] Negotiation successful
0807 07:01:25.843469 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2104,"thread_start_us":1966,"threads_started":1}
I20260807 07:01:25.844584 23454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:25.828289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39208 (local address 127.21.83.2:42537)
0807 07:01:25.829538 (+ 1249us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:25.829546 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:25.830226 (+ 680us) server_negotiation.cc:407] Connection header received
0807 07:01:25.830324 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:25.830330 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:25.831878 (+ 1548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:25.832079 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:01:25.833062 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:25.835905 (+ 2843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:25.839833 (+ 3928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:25.841751 (+ 1918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:25.841916 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:25.842155 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:25.843438 (+ 1283us) server_negotiation.cc:299] Negotiation successful
0807 07:01:25.844273 (+ 835us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1090,"thread_start_us":982,"threads_started":1}
I20260807 07:01:25.846841 22926 tablet_service.cc:1467] Processing CreateTablet for tablet 6cc5f6de482f4785b2b8385eec0ee26e (DEFAULT_TABLE table=test-table [id=e4453adf04274e65855cc8e700e72407]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:01:25.848548 22926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cc5f6de482f4785b2b8385eec0ee26e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:01:25.848230 23132 tablet_service.cc:1467] Processing CreateTablet for tablet 6cc5f6de482f4785b2b8385eec0ee26e (DEFAULT_TABLE table=test-table [id=e4453adf04274e65855cc8e700e72407]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:01:25.849804 23132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cc5f6de482f4785b2b8385eec0ee26e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:01:25.851708 23449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:25.830180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.3:37641 (local address 127.0.0.1:38210)
0807 07:01:25.842710 (+ 12530us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:25.842731 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:25.842846 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:25.843891 (+ 1045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:25.843896 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:25.843929 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:01:25.844107 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:25.844120 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:25.846557 (+ 2437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:25.846562 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:25.848357 (+ 1795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:25.848378 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:25.850845 (+ 2467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:25.850851 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:25.851048 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:25.851083 (+ 35us) client_negotiation.cc:769] Sending connection context
0807 07:01:25.851189 (+ 106us) client_negotiation.cc:240] Negotiation successful
0807 07:01:25.851351 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11357,"thread_start_us":12052,"threads_started":1}
I20260807 07:01:25.851818 23456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:25.830303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38210 (local address 127.21.83.3:37641)
0807 07:01:25.842905 (+ 12602us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:25.842917 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:25.842956 (+ 39us) server_negotiation.cc:407] Connection header received
0807 07:01:25.843071 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:25.843080 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:25.843375 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:25.844580 (+ 1205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:01:25.845084 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:25.846358 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:25.848561 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:25.850276 (+ 1715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:25.850432 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:25.850692 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:25.851281 (+ 589us) server_negotiation.cc:299] Negotiation successful
0807 07:01:25.851464 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12418,"thread_start_us":12242,"threads_started":1}
I20260807 07:01:25.856323 23335 tablet_service.cc:1467] Processing CreateTablet for tablet 6cc5f6de482f4785b2b8385eec0ee26e (DEFAULT_TABLE table=test-table [id=e4453adf04274e65855cc8e700e72407]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:01:25.858022 23335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cc5f6de482f4785b2b8385eec0ee26e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:01:25.863422 23457 tablet_bootstrap.cc:492] T 6cc5f6de482f4785b2b8385eec0ee26e P b49b2228389944659526ae92e017a572: Bootstrap starting.
I20260807 07:01:25.865475 23459 tablet_bootstrap.cc:492] T 6cc5f6de482f4785b2b8385eec0ee26e P 3ce3d28be1cd440f85cb208a81b04393: Bootstrap starting.
I20260807 07:01:25.867201 23457 tablet_bootstrap.cc:654] T 6cc5f6de482f4785b2b8385eec0ee26e P b49b2228389944659526ae92e017a572: Neither blocks nor log segments found. Creating new log.
I20260807 07:01:25.868733 23459 tablet_bootstrap.cc:654] T 6cc5f6de482f4785b2b8385eec0ee26e P 3ce3d28be1cd440f85cb208a81b04393: Neither blocks nor log segments found. Creating new log.
I20260807 07:01:25.869202 23457 log.cc:824] T 6cc5f6de482f4785b2b8385eec0ee26e P b49b2228389944659526ae92e017a572: Log is configured to *not* fsync() on all Append() calls
I20260807 07:01:25.869748 23459 log.cc:824] T 6cc5f6de482f4785b2b8385eec0ee26e P 3ce3d28be1cd440f85cb208a81b04393: Log is configured to *not* fsync() on all Append() calls
I20260807 07:01:25.872257 23457 tablet_bootstrap.cc:492] T 6cc5f6de482f4785b2b8385eec0ee26e P b49b2228389944659526ae92e017a572: No bootstrap required, opened a new log
I20260807 07:01:25.872404 23464 tablet_bootstrap.cc:492] T 6cc5f6de482f4785b2b8385eec0ee26e P f6275135901a45b492bd4643e41ac801: Bootstrap starting.
I20260807 07:01:25.872649 23457 ts_tablet_manager.cc:1397] T 6cc5f6de482f4785b2b8385eec0ee26e P b49b2228389944659526ae92e017a572: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20260807 07:01:25.873190 23459 tablet_bootstrap.cc:492] T 6cc5f6de482f4785b2b8385eec0ee26e P 3ce3d28be1cd440f85cb208a81b04393: No bootstrap required, opened a new log
I20260807 07:01:25.873458 23459 ts_tablet_manager.cc:1397] T 6cc5f6de482f4785b2b8385eec0ee26e P 3ce3d28be1cd440f85cb208a81b04393: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260807 07:01:25.875203 23464 tablet_bootstrap.cc:654] T 6cc5f6de482f4785b2b8385eec0ee26e P f6275135901a45b492bd4643e41ac801: Neither blocks nor log segments found. Creating new log.
I20260807 07:01:25.876199 23464 log.cc:824] T 6cc5f6de482f4785b2b8385eec0ee26e P f6275135901a45b492bd4643e41ac801: Log is configured to *not* fsync() on all Append() calls
I20260807 07:01:25.879269 23464 tablet_bootstrap.cc:492] T 6cc5f6de482f4785b2b8385eec0ee26e P f6275135901a45b492bd4643e41ac801: No bootstrap required, opened a new log
I20260807 07:01:25.879510 23464 ts_tablet_manager.cc:1397] T 6cc5f6de482f4785b2b8385eec0ee26e P f6275135901a45b492bd4643e41ac801: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260807 07:01:25.883754 23459 raft_consensus.cc:348] T 6cc5f6de482f4785b2b8385eec0ee26e P 3ce3d28be1cd440f85cb208a81b04393 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ce3d28be1cd440f85cb208a81b04393" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 46291 } } peers { permanent_uuid: "f6275135901a45b492bd4643e41ac801" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 37641 } } peers { permanent_uuid: "b49b2228389944659526ae92e017a572" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 42537 } }
I20260807 07:01:25.883754 23457 raft_consensus.cc:348] T 6cc5f6de482f4785b2b8385eec0ee26e P b49b2228389944659526ae92e017a572 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ce3d28be1cd440f85cb208a81b04393" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 46291 } } peers { permanent_uuid: "f6275135901a45b492bd4643e41ac801" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 37641 } } peers { permanent_uuid: "b49b2228389944659526ae92e017a572" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 42537 } }
I20260807 07:01:25.884207 23459 raft_consensus.cc:374] T 6cc5f6de482f4785b2b8385eec0ee26e P 3ce3d28be1cd440f85cb208a81b04393 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:01:25.884207 23457 raft_consensus.cc:374] T 6cc5f6de482f4785b2b8385eec0ee26e P b49b2228389944659526ae92e017a572 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:01:25.884310 23457 raft_consensus.cc:729] T 6cc5f6de482f4785b2b8385eec0ee26e P b49b2228389944659526ae92e017a572 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b49b2228389944659526ae92e017a572, State: Initialized, Role: FOLLOWER
I20260807 07:01:25.884336 23459 raft_consensus.cc:729] T 6cc5f6de482f4785b2b8385eec0ee26e P 3ce3d28be1cd440f85cb208a81b04393 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ce3d28be1cd440f85cb208a81b04393, State: Initialized, Role: FOLLOWER
I20260807 07:01:25.884987 23459 consensus_queue.cc:260] T 6cc5f6de482f4785b2b8385eec0ee26e P 3ce3d28be1cd440f85cb208a81b04393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ce3d28be1cd440f85cb208a81b04393" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 46291 } } peers { permanent_uuid: "f6275135901a45b492bd4643e41ac801" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 37641 } } peers { permanent_uuid: "b49b2228389944659526ae92e017a572" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 42537 } }
I20260807 07:01:25.885011 23457 consensus_queue.cc:260] T 6cc5f6de482f4785b2b8385eec0ee26e P b49b2228389944659526ae92e017a572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ce3d28be1cd440f85cb208a81b04393" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 46291 } } peers { permanent_uuid: "f6275135901a45b492bd4643e41ac801" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 37641 } } peers { permanent_uuid: "b49b2228389944659526ae92e017a572" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 42537 } }
I20260807 07:01:25.886178 23230 heartbeater.cc:503] Master 127.21.83.62:38191 was elected leader, sending a full tablet report...
I20260807 07:01:25.886932 23457 ts_tablet_manager.cc:1428] T 6cc5f6de482f4785b2b8385eec0ee26e P b49b2228389944659526ae92e017a572: Time spent starting tablet: real 0.014s user 0.007s sys 0.006s
I20260807 07:01:25.887787 23464 raft_consensus.cc:348] T 6cc5f6de482f4785b2b8385eec0ee26e P f6275135901a45b492bd4643e41ac801 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ce3d28be1cd440f85cb208a81b04393" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 46291 } } peers { permanent_uuid: "f6275135901a45b492bd4643e41ac801" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 37641 } } peers { permanent_uuid: "b49b2228389944659526ae92e017a572" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 42537 } }
I20260807 07:01:25.888304 23464 raft_consensus.cc:374] T 6cc5f6de482f4785b2b8385eec0ee26e P f6275135901a45b492bd4643e41ac801 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:01:25.888464 23464 raft_consensus.cc:729] T 6cc5f6de482f4785b2b8385eec0ee26e P f6275135901a45b492bd4643e41ac801 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6275135901a45b492bd4643e41ac801, State: Initialized, Role: FOLLOWER
I20260807 07:01:25.888795 23459 ts_tablet_manager.cc:1428] T 6cc5f6de482f4785b2b8385eec0ee26e P 3ce3d28be1cd440f85cb208a81b04393: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
I20260807 07:01:25.889294 23464 consensus_queue.cc:260] T 6cc5f6de482f4785b2b8385eec0ee26e P f6275135901a45b492bd4643e41ac801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ce3d28be1cd440f85cb208a81b04393" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 46291 } } peers { permanent_uuid: "f6275135901a45b492bd4643e41ac801" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 37641 } } peers { permanent_uuid: "b49b2228389944659526ae92e017a572" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 42537 } }
I20260807 07:01:25.890532 23428 heartbeater.cc:503] Master 127.21.83.62:38191 was elected leader, sending a full tablet report...
I20260807 07:01:25.891111 23464 ts_tablet_manager.cc:1428] T 6cc5f6de482f4785b2b8385eec0ee26e P f6275135901a45b492bd4643e41ac801: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
W20260807 07:01:25.997097 23025 tablet.cc:2369] T 6cc5f6de482f4785b2b8385eec0ee26e P 3ce3d28be1cd440f85cb208a81b04393: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 07:01:26.001533 23231 tablet.cc:2369] T 6cc5f6de482f4785b2b8385eec0ee26e P b49b2228389944659526ae92e017a572: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 07:01:26.002753 23467 raft_consensus.cc:482] T 6cc5f6de482f4785b2b8385eec0ee26e P 3ce3d28be1cd440f85cb208a81b04393 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:01:26.003042 23467 raft_consensus.cc:504] T 6cc5f6de482f4785b2b8385eec0ee26e P 3ce3d28be1cd440f85cb208a81b04393 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ce3d28be1cd440f85cb208a81b04393" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 46291 } } peers { permanent_uuid: "f6275135901a45b492bd4643e41ac801" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 37641 } } peers { permanent_uuid: "b49b2228389944659526ae92e017a572" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 42537 } }
I20260807 07:01:26.004794 23467 leader_election.cc:288] T 6cc5f6de482f4785b2b8385eec0ee26e P 3ce3d28be1cd440f85cb208a81b04393 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6275135901a45b492bd4643e41ac801 (127.21.83.3:37641), b49b2228389944659526ae92e017a572 (127.21.83.2:42537)
W20260807 07:01:26.007910 23430 tablet.cc:2369] T 6cc5f6de482f4785b2b8385eec0ee26e P f6275135901a45b492bd4643e41ac801: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 07:01:26.012663 23454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:26.005472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:54241 (local address 127.21.83.2:42537)
0807 07:01:26.005690 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:26.005697 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:26.006348 (+ 651us) server_negotiation.cc:407] Connection header received
0807 07:01:26.006480 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:26.006486 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:26.006562 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:26.006648 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:01:26.007374 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:26.008287 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:26.011004 (+ 2717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:26.011804 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:26.011914 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:26.011988 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:26.012274 (+ 286us) server_negotiation.cc:299] Negotiation successful
0807 07:01:26.012409 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 07:01:26.012748 23471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:26.005297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.2:42537 (local address 127.21.83.1:54241)
0807 07:01:26.006243 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:26.006262 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:26.006373 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:26.006667 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:26.006672 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:26.006698 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:01:26.007141 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:26.007156 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:26.008421 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:26.008426 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:26.010846 (+ 2420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:26.010862 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:26.011984 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:26.011988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:26.012105 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:26.012130 (+ 25us) client_negotiation.cc:769] Sending connection context
0807 07:01:26.012213 (+ 83us) client_negotiation.cc:240] Negotiation successful
0807 07:01:26.012340 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":847,"mutex_wait_us":163,"thread_start_us":467,"threads_started":1}
I20260807 07:01:26.013690 23456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:26.005289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:49935 (local address 127.21.83.3:37641)
0807 07:01:26.005505 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:26.005511 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:26.006242 (+ 731us) server_negotiation.cc:407] Connection header received
0807 07:01:26.006482 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:26.006487 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:26.006562 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:26.006623 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:01:26.007422 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:26.008537 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:26.012060 (+ 3523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:26.012876 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:26.013008 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:26.013131 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:26.013385 (+ 254us) server_negotiation.cc:299] Negotiation successful
0807 07:01:26.013510 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260807 07:01:26.013774 23472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:26.005111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.3:37641 (local address 127.21.83.1:49935)
0807 07:01:26.006125 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:26.006141 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:26.006300 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:26.006853 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:26.006859 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:26.006905 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:01:26.007288 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:26.007299 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:26.008670 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:26.008675 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:26.010848 (+ 2173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:26.010861 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:26.013101 (+ 2240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:26.013107 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:26.013233 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:26.013257 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 07:01:26.013331 (+ 74us) client_negotiation.cc:240] Negotiation successful
0807 07:01:26.013418 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":888,"mutex_wait_us":20,"thread_start_us":773,"threads_started":1}
I20260807 07:01:26.014051 23162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cc5f6de482f4785b2b8385eec0ee26e" candidate_uuid: "3ce3d28be1cd440f85cb208a81b04393" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b49b2228389944659526ae92e017a572" is_pre_election: true
I20260807 07:01:26.014643 23162 raft_consensus.cc:2454] T 6cc5f6de482f4785b2b8385eec0ee26e P b49b2228389944659526ae92e017a572 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ce3d28be1cd440f85cb208a81b04393 in term 0.
I20260807 07:01:26.014835 23365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cc5f6de482f4785b2b8385eec0ee26e" candidate_uuid: "3ce3d28be1cd440f85cb208a81b04393" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6275135901a45b492bd4643e41ac801" is_pre_election: true
I20260807 07:01:26.015292 23365 raft_consensus.cc:2454] T 6cc5f6de482f4785b2b8385eec0ee26e P f6275135901a45b492bd4643e41ac801 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ce3d28be1cd440f85cb208a81b04393 in term 0.
I20260807 07:01:26.015288 22859 leader_election.cc:302] T 6cc5f6de482f4785b2b8385eec0ee26e P 3ce3d28be1cd440f85cb208a81b04393 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ce3d28be1cd440f85cb208a81b04393, b49b2228389944659526ae92e017a572; no voters:
I20260807 07:01:26.015661 23467 raft_consensus.cc:2781] T 6cc5f6de482f4785b2b8385eec0ee26e P 3ce3d28be1cd440f85cb208a81b04393 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 07:01:26.015805 23467 raft_consensus.cc:482] T 6cc5f6de482f4785b2b8385eec0ee26e P 3ce3d28be1cd440f85cb208a81b04393 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:01:26.015906 23467 raft_consensus.cc:3037] T 6cc5f6de482f4785b2b8385eec0ee26e P 3ce3d28be1cd440f85cb208a81b04393 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:01:26.018190 23467 raft_consensus.cc:504] T 6cc5f6de482f4785b2b8385eec0ee26e P 3ce3d28be1cd440f85cb208a81b04393 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ce3d28be1cd440f85cb208a81b04393" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 46291 } } peers { permanent_uuid: "f6275135901a45b492bd4643e41ac801" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 37641 } } peers { permanent_uuid: "b49b2228389944659526ae92e017a572" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 42537 } }
I20260807 07:01:26.018821 23467 leader_election.cc:288] T 6cc5f6de482f4785b2b8385eec0ee26e P 3ce3d28be1cd440f85cb208a81b04393 [CANDIDATE]: Term 1 election: Requested vote from peers f6275135901a45b492bd4643e41ac801 (127.21.83.3:37641), b49b2228389944659526ae92e017a572 (127.21.83.2:42537)
I20260807 07:01:26.019301 23365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cc5f6de482f4785b2b8385eec0ee26e" candidate_uuid: "3ce3d28be1cd440f85cb208a81b04393" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6275135901a45b492bd4643e41ac801"
I20260807 07:01:26.019318 23162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cc5f6de482f4785b2b8385eec0ee26e" candidate_uuid: "3ce3d28be1cd440f85cb208a81b04393" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b49b2228389944659526ae92e017a572"
I20260807 07:01:26.019484 23365 raft_consensus.cc:3037] T 6cc5f6de482f4785b2b8385eec0ee26e P f6275135901a45b492bd4643e41ac801 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:01:26.019526 23162 raft_consensus.cc:3037] T 6cc5f6de482f4785b2b8385eec0ee26e P b49b2228389944659526ae92e017a572 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:01:26.021795 23365 raft_consensus.cc:2454] T 6cc5f6de482f4785b2b8385eec0ee26e P f6275135901a45b492bd4643e41ac801 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ce3d28be1cd440f85cb208a81b04393 in term 1.
I20260807 07:01:26.021852 23162 raft_consensus.cc:2454] T 6cc5f6de482f4785b2b8385eec0ee26e P b49b2228389944659526ae92e017a572 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ce3d28be1cd440f85cb208a81b04393 in term 1.
I20260807 07:01:26.022209 22858 leader_election.cc:302] T 6cc5f6de482f4785b2b8385eec0ee26e P 3ce3d28be1cd440f85cb208a81b04393 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ce3d28be1cd440f85cb208a81b04393, f6275135901a45b492bd4643e41ac801; no voters:
I20260807 07:01:26.022507 23467 raft_consensus.cc:2781] T 6cc5f6de482f4785b2b8385eec0ee26e P 3ce3d28be1cd440f85cb208a81b04393 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:01:26.022943 23467 raft_consensus.cc:686] T 6cc5f6de482f4785b2b8385eec0ee26e P 3ce3d28be1cd440f85cb208a81b04393 [term 1 LEADER]: Becoming Leader. State: Replica: 3ce3d28be1cd440f85cb208a81b04393, State: Running, Role: LEADER
I20260807 07:01:26.023473 23467 consensus_queue.cc:237] T 6cc5f6de482f4785b2b8385eec0ee26e P 3ce3d28be1cd440f85cb208a81b04393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ce3d28be1cd440f85cb208a81b04393" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 46291 } } peers { permanent_uuid: "f6275135901a45b492bd4643e41ac801" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 37641 } } peers { permanent_uuid: "b49b2228389944659526ae92e017a572" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 42537 } }
I20260807 07:01:26.027702 22746 catalog_manager.cc:5495] T 6cc5f6de482f4785b2b8385eec0ee26e P 3ce3d28be1cd440f85cb208a81b04393 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ce3d28be1cd440f85cb208a81b04393 (127.21.83.1). New cstate: current_term: 1 leader_uuid: "3ce3d28be1cd440f85cb208a81b04393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ce3d28be1cd440f85cb208a81b04393" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 46291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6275135901a45b492bd4643e41ac801" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 37641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b49b2228389944659526ae92e017a572" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 42537 } health_report { overall_health: UNKNOWN } } }
Aug 07 07:01:26 dist-test-slave-ncp9 krb5kdc[22674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.1@KRBTEST.COM
I20260807 07:01:26.082592 23450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:26.068027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40932 (local address 127.21.83.1:46291)
0807 07:01:26.068249 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:26.068257 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:26.068828 (+ 571us) server_negotiation.cc:407] Connection header received
0807 07:01:26.069024 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:26.069030 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:26.069126 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:26.069214 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:26.070717 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:26.071424 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:26.072698 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:26.073139 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:26.073249 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:26.077236 (+ 3987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:26.077252 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:26.077267 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:26.077305 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:26.079941 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:26.080546 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:26.080553 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:26.080555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:26.080713 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:26.081106 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:26.081109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:26.081111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:26.081460 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:26.081810 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:26.082036 (+ 226us) server_negotiation.cc:299] Negotiation successful
0807 07:01:26.082312 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
Aug 07 07:01:26 dist-test-slave-ncp9 krb5kdc[22674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086086, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 07:01:26 dist-test-slave-ncp9 krb5kdc[22674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086086, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.83.1@KRBTEST.COM
I20260807 07:01:26.131345 23450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:26.118461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40948 (local address 127.21.83.1:46291)
0807 07:01:26.118661 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:26.118668 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:26.119004 (+ 336us) server_negotiation.cc:407] Connection header received
0807 07:01:26.119185 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:26.119189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:26.119257 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:26.119340 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:26.120680 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:26.121462 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:26.122743 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:26.123152 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:26.123267 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:26.126988 (+ 3721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:26.126993 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:26.126996 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:26.127030 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:26.129370 (+ 2340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:26.129894 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:26.129898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:26.129900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:26.130018 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:26.130286 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:26.130289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:26.130291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:26.130528 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:26.130789 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:26.130903 (+ 114us) server_negotiation.cc:299] Negotiation successful
0807 07:01:26.131087 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Aug 07 07:01:26 dist-test-slave-ncp9 krb5kdc[22674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 07:01:26 dist-test-slave-ncp9 krb5kdc[22674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.1@KRBTEST.COM
I20260807 07:01:26.178037 23450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:26.163752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40950 (local address 127.21.83.1:46291)
0807 07:01:26.163946 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:26.163952 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:26.164293 (+ 341us) server_negotiation.cc:407] Connection header received
0807 07:01:26.164473 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:26.164478 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:26.164562 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:26.164650 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:26.166127 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:26.166928 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:26.168203 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:26.168644 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:26.168765 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:26.172802 (+ 4037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:26.172807 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:26.172811 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:26.172855 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:26.175639 (+ 2784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:26.176347 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:26.176351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:26.176354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:26.176487 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:26.176787 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:26.176789 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:26.176791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:26.177126 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:26.177345 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:26.177572 (+ 227us) server_negotiation.cc:299] Negotiation successful
0807 07:01:26.177766 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260807 07:01:26.179853 22896 tablet_service.cc:2908] Rejecting scan request for tablet 6cc5f6de482f4785b2b8385eec0ee26e: Uninitialized: clean time has not yet been initialized
I20260807 07:01:26.183187 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 22833
I20260807 07:01:26.196415 23450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:26.184098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40958 (local address 127.21.83.1:46291)
0807 07:01:26.184303 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:26.184311 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:26.184659 (+ 348us) server_negotiation.cc:407] Connection header received
0807 07:01:26.184783 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:26.184787 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:26.184872 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:26.185060 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:26.186219 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:26.186923 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:26.188108 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:26.188541 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:26.188648 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:26.190903 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:26.190926 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:26.190931 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:26.190978 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:26.193853 (+ 2875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:26.194539 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:26.194543 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:26.194546 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:26.194681 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:26.195024 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:26.195027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:26.195029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:26.195343 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:26.195552 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:26.195850 (+ 298us) server_negotiation.cc:299] Negotiation successful
0807 07:01:26.196059 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 07:01:26.197261 23016 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:01:26.288861 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 22833
I20260807 07:01:26.311769 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 23034
Aug 07 07:01:26 dist-test-slave-ncp9 krb5kdc[22674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.2@KRBTEST.COM
I20260807 07:01:26.326110 23454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:26.312531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39220 (local address 127.21.83.2:42537)
0807 07:01:26.312706 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:26.312711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:26.312730 (+ 19us) server_negotiation.cc:407] Connection header received
0807 07:01:26.312983 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:26.312987 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:26.313102 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:26.313195 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:26.314441 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:26.315237 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:26.316495 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:26.317019 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:26.317118 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:26.320931 (+ 3813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:26.320951 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:26.320969 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:26.321010 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:26.323539 (+ 2529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:26.324157 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:26.324163 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:26.324166 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:26.324291 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:26.324571 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:26.324575 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:26.324576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:26.325139 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:26.325378 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:26.325613 (+ 235us) server_negotiation.cc:299] Negotiation successful
0807 07:01:26.325779 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 07:01:26.326737 23222 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:01:26.415359 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 23034
I20260807 07:01:26.432912 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 23239
Aug 07 07:01:26 dist-test-slave-ncp9 krb5kdc[22674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.3@KRBTEST.COM
I20260807 07:01:26.447222 23456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:26.433723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38216 (local address 127.21.83.3:37641)
0807 07:01:26.433884 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:26.433888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:26.433908 (+ 20us) server_negotiation.cc:407] Connection header received
0807 07:01:26.434023 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:26.434027 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:26.434092 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:26.434179 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:26.435467 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:26.436205 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:26.437621 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:26.438020 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:26.438113 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:26.442083 (+ 3970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:26.442100 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:26.442116 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:26.442158 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:26.444682 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:26.445390 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:26.445398 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:26.445400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:26.445546 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:26.445935 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:26.445937 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:26.445939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:26.446293 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:26.446521 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:26.446769 (+ 248us) server_negotiation.cc:299] Negotiation successful
0807 07:01:26.446944 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 07:01:26.447921 23421 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:01:26.534273 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 23239
I20260807 07:01:26.556325 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 22697
I20260807 07:01:26.557596 22790 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:01:26.668532 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 22697
2026-08-07T07:01:26Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4366 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 07:01:26 dist-test-slave-ncp9 krb5kdc[23502](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 07:01:26 dist-test-slave-ncp9 krb5kdc[23502](info): set up 2 sockets
Aug 07 07:01:26 dist-test-slave-ncp9 krb5kdc[23502](info): commencing operation
krb5kdc: starting...
W20260807 07:01:27.847533 21836 mini_kdc.cc:121] Time spent starting KDC: real 1.131s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Aug 07 07:01:27 dist-test-slave-ncp9 krb5kdc[23502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T07:01:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:01:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
I20260807 07:01:28.056772 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:33131
--webserver_interface=127.21.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:45239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:33131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:01:28.387509 23521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:01:28.387863 23521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:01:28.387995 23521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:01:28.395442 23521 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:01:28.395574 23521 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260807 07:01:28.395640 23521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:01:28.395735 23521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:01:28.395802 23521 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:01:28.395885 23521 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:01:28.406090 23521 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:45239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:33131
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:33131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:01:28.407297 23521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:01:28.408573 23521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:01:28.419615 23527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:28.419570 23528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:28.419677 23531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:01:28.420632 23521 server_base.cc:1034] running on GCE node
I20260807 07:01:28.421387 23521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:01:28.422701 23521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:01:28.423949 23521 hybrid_clock.cc:648] HybridClock initialized: now 1786086088423876 us; error 35 us; skew 500 ppm
I20260807 07:01:28.424285 23521 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:01:28 dist-test-slave-ncp9 krb5kdc[23502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086088, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:01:28.427399 23521 init.cc:377] Logged in from keytab as kudu/127.21.83.62@KRBTEST.COM (short username kudu)
I20260807 07:01:28.428854 23521 webserver.cc:460] Webserver started at http://127.21.83.62:37111/ using document root <none> and password file <none>
I20260807 07:01:28.429827 23521 fs_manager.cc:359] Metadata directory not provided
I20260807 07:01:28.429957 23521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:01:28.430286 23521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:01:28.433321 23521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "e5bf836233d747769243d412d635e83f"
format_stamp: "Formatted at 2026-08-07 07:01:28 on dist-test-slave-ncp9"
server_key: "8a3218e83861447d34a51baee97a8bfc"
server_key_iv: "f6c9ce532921d209d5b31ac96f2049a9"
server_key_version: "encryptionkey@0"
I20260807 07:01:28.434074 23521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "e5bf836233d747769243d412d635e83f"
format_stamp: "Formatted at 2026-08-07 07:01:28 on dist-test-slave-ncp9"
server_key: "8a3218e83861447d34a51baee97a8bfc"
server_key_iv: "f6c9ce532921d209d5b31ac96f2049a9"
server_key_version: "encryptionkey@0"
I20260807 07:01:28.439265 23521 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260807 07:01:28.443287 23542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:28.444697 23521 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 07:01:28.444856 23521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "e5bf836233d747769243d412d635e83f"
format_stamp: "Formatted at 2026-08-07 07:01:28 on dist-test-slave-ncp9"
server_key: "8a3218e83861447d34a51baee97a8bfc"
server_key_iv: "f6c9ce532921d209d5b31ac96f2049a9"
server_key_version: "encryptionkey@0"
I20260807 07:01:28.445125 23521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:01:28.461022 23521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:01:28.465691 23521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:01:28.465936 23521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:01:28.487154 23521 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.62:33131
I20260807 07:01:28.487178 23629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.62:33131 every 8 connection(s)
I20260807 07:01:28.488487 23521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:01:28.492630 23637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:01:28.498127 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 23521
I20260807 07:01:28.498387 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:01:28.499214 21836 external_mini_cluster.cc:1428] Setting key a01832c2124b6e571e8f3184c350a1d6
I20260807 07:01:28.503530 23637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5bf836233d747769243d412d635e83f: Bootstrap starting.
I20260807 07:01:28.506907 23637 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5bf836233d747769243d412d635e83f: Neither blocks nor log segments found. Creating new log.
I20260807 07:01:28.507982 23637 log.cc:824] T 00000000000000000000000000000000 P e5bf836233d747769243d412d635e83f: Log is configured to *not* fsync() on all Append() calls
I20260807 07:01:28.510780 23637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5bf836233d747769243d412d635e83f: No bootstrap required, opened a new log
Aug 07 07:01:28 dist-test-slave-ncp9 krb5kdc[23502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:01:28.519392 23637 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5bf836233d747769243d412d635e83f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5bf836233d747769243d412d635e83f" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 33131 } }
I20260807 07:01:28.519786 23637 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5bf836233d747769243d412d635e83f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:01:28.519886 23637 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5bf836233d747769243d412d635e83f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5bf836233d747769243d412d635e83f, State: Initialized, Role: FOLLOWER
I20260807 07:01:28.520558 23637 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5bf836233d747769243d412d635e83f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5bf836233d747769243d412d635e83f" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 33131 } }
I20260807 07:01:28.520869 23637 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5bf836233d747769243d412d635e83f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:01:28.521118 23637 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5bf836233d747769243d412d635e83f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:01:28.521281 23637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5bf836233d747769243d412d635e83f [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:01:28.522207 23640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:28.501294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44944 (local address 127.21.83.62:33131)
0807 07:01:28.502075 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:28.502086 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:28.502126 (+ 40us) server_negotiation.cc:407] Connection header received
0807 07:01:28.503803 (+ 1677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:28.503824 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:28.504172 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:28.504453 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:28.506217 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:28.507577 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:28.509117 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:28.509733 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:28.509856 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:28.514671 (+ 4815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:28.514679 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:28.514694 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:28.514746 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:28.517881 (+ 3135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:28.518714 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:28.518722 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:28.518728 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:28.518911 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:28.519349 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:28.519353 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:28.519356 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:28.520038 (+ 682us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:28.520322 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:28.520699 (+ 377us) server_negotiation.cc:299] Negotiation successful
0807 07:01:28.521152 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":364,"threads_started":1}
I20260807 07:01:28.523811 23637 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5bf836233d747769243d412d635e83f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5bf836233d747769243d412d635e83f" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 33131 } }
I20260807 07:01:28.524406 23637 leader_election.cc:302] T 00000000000000000000000000000000 P e5bf836233d747769243d412d635e83f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5bf836233d747769243d412d635e83f; no voters:
I20260807 07:01:28.525097 23637 leader_election.cc:288] T 00000000000000000000000000000000 P e5bf836233d747769243d412d635e83f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 07:01:28.525272 23642 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5bf836233d747769243d412d635e83f [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:01:28.526088 23642 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5bf836233d747769243d412d635e83f [term 1 LEADER]: Becoming Leader. State: Replica: e5bf836233d747769243d412d635e83f, State: Running, Role: LEADER
I20260807 07:01:28.526613 23642 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5bf836233d747769243d412d635e83f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5bf836233d747769243d412d635e83f" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 33131 } }
I20260807 07:01:28.527208 23637 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5bf836233d747769243d412d635e83f [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:01:28.529433 23644 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5bf836233d747769243d412d635e83f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5bf836233d747769243d412d635e83f. Latest consensus state: current_term: 1 leader_uuid: "e5bf836233d747769243d412d635e83f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5bf836233d747769243d412d635e83f" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 33131 } } }
I20260807 07:01:28.529852 23644 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5bf836233d747769243d412d635e83f [sys.catalog]: This master's current role is: LEADER
I20260807 07:01:28.529450 23643 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5bf836233d747769243d412d635e83f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5bf836233d747769243d412d635e83f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5bf836233d747769243d412d635e83f" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 33131 } } }
I20260807 07:01:28.533156 23643 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5bf836233d747769243d412d635e83f [sys.catalog]: This master's current role is: LEADER
I20260807 07:01:28.534298 23649 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:01:28.539101 23649 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:01:28.549068 23649 catalog_manager.cc:1347] Generated new cluster ID: 015aa457f8c84dd79f948c3d7a87e6ec
I20260807 07:01:28.549178 23649 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:01:28.584141 23649 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:01:28.586025 23649 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:01:28.594934 23649 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5bf836233d747769243d412d635e83f: Generated new TSK 0
I20260807 07:01:28.595901 23649 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.83.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab.
Entry for principal kudu/127.21.83.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab.
I20260807 07:01:28.643647 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.1:0
--local_ip_for_outbound_sockets=127.21.83.1
--webserver_interface=127.21.83.1
--webserver_port=0
--tserver_master_addrs=127.21.83.62:33131
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab
--principal=kudu/127.21.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:45239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:01:28.968194 23672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:01:28.968473 23672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:01:28.968596 23672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:01:28.976061 23672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:01:28.976209 23672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:01:28.976320 23672 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 07:01:28.976437 23672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.1
I20260807 07:01:28.986867 23672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:45239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab
--principal=kudu/127.21.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.83.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.83.62:33131
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.1
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:01:28.988253 23672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:01:28.989722 23672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:01:29.004136 23683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:29.004050 23687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:29.004072 23691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:01:29.005175 23672 server_base.cc:1034] running on GCE node
I20260807 07:01:29.005856 23672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:01:29.006879 23672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:01:29.008118 23672 hybrid_clock.cc:648] HybridClock initialized: now 1786086089008044 us; error 47 us; skew 500 ppm
I20260807 07:01:29.008438 23672 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:01:29 dist-test-slave-ncp9 krb5kdc[23502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086089, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:01:29.011737 23672 init.cc:377] Logged in from keytab as kudu/127.21.83.1@KRBTEST.COM (short username kudu)
I20260807 07:01:29.013403 23672 webserver.cc:460] Webserver started at http://127.21.83.1:33249/ using document root <none> and password file <none>
I20260807 07:01:29.014133 23672 fs_manager.cc:359] Metadata directory not provided
I20260807 07:01:29.014241 23672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:01:29.014568 23672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:01:29.017728 23672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-0/data/instance:
uuid: "3d6807a83f544e32ae7874e242bf51ee"
format_stamp: "Formatted at 2026-08-07 07:01:29 on dist-test-slave-ncp9"
server_key: "3a6575d829a23442aa940b9a4c5d85c2"
server_key_iv: "3cc05581c1eb1d1d0d7f946e5150efae"
server_key_version: "encryptionkey@0"
I20260807 07:01:29.018496 23672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-0/wal/instance:
uuid: "3d6807a83f544e32ae7874e242bf51ee"
format_stamp: "Formatted at 2026-08-07 07:01:29 on dist-test-slave-ncp9"
server_key: "3a6575d829a23442aa940b9a4c5d85c2"
server_key_iv: "3cc05581c1eb1d1d0d7f946e5150efae"
server_key_version: "encryptionkey@0"
I20260807 07:01:29.023586 23672 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 07:01:29.027124 23704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:29.028478 23672 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 07:01:29.028651 23672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-0/wal
uuid: "3d6807a83f544e32ae7874e242bf51ee"
format_stamp: "Formatted at 2026-08-07 07:01:29 on dist-test-slave-ncp9"
server_key: "3a6575d829a23442aa940b9a4c5d85c2"
server_key_iv: "3cc05581c1eb1d1d0d7f946e5150efae"
server_key_version: "encryptionkey@0"
I20260807 07:01:29.028842 23672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:01:29.046103 23672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:01:29.050375 23672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:01:29.050616 23672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:01:29.053350 23672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:01:29.053457 23672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:29.053622 23672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:01:29.053702 23672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 07:01:29 dist-test-slave-ncp9 krb5kdc[23502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086089, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.1@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:01:29.080140 23718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:29.053974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:39343 (local address 127.21.83.62:33131)
0807 07:01:29.054663 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:29.054668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:29.055835 (+ 1167us) server_negotiation.cc:407] Connection header received
0807 07:01:29.058340 (+ 2505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:29.058348 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:29.058453 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:29.058588 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:29.061331 (+ 2743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:29.062541 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:29.066725 (+ 4184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:29.067463 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:29.067863 (+ 400us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:29.073253 (+ 5390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:29.073261 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:29.073266 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:29.073321 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:29.076146 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:29.077048 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:29.077055 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:29.077058 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:29.077197 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:29.077775 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:29.077781 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:29.077784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:29.078250 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:29.078425 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:29.079476 (+ 1051us) server_negotiation.cc:299] Negotiation successful
0807 07:01:29.079762 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":451,"threads_started":1}
I20260807 07:01:29.081858 23719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:29.054699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:33131 (local address 127.21.83.1:39343)
0807 07:01:29.055636 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:29.055689 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:29.057969 (+ 2280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:29.058862 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:29.058870 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:29.059786 (+ 916us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:01:29.061025 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:29.061068 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:29.062867 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:29.062873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:29.066407 (+ 3534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:29.066444 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:29.068288 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:29.068296 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:29.068562 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:29.069429 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:01:29.069469 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:01:29.072868 (+ 3399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:01:29.076337 (+ 3469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:29.076348 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:29.076368 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:29.076874 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:29.077319 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:29.077325 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:29.077329 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:29.077577 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:29.078419 (+ 842us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:01:29.078428 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:01:29.078936 (+ 508us) client_negotiation.cc:769] Sending connection context
0807 07:01:29.079334 (+ 398us) client_negotiation.cc:240] Negotiation successful
0807 07:01:29.080028 (+ 694us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":408,"threads_started":1}
I20260807 07:01:29.106247 23672 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.1:42307
I20260807 07:01:29.106259 23901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.1:42307 every 8 connection(s)
I20260807 07:01:29.107507 23854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:29.090441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:33131 (local address 127.21.83.1:46063)
0807 07:01:29.091367 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:29.091390 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:29.091592 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:29.092063 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:29.092068 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:29.092616 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:01:29.093004 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:29.093022 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:29.094563 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:29.094567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:29.096142 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:29.096158 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:29.097239 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:29.097246 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:29.097430 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:29.098190 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:01:29.098209 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:01:29.099034 (+ 825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:01:29.104345 (+ 5311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:29.104352 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:29.104356 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:29.105008 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:29.105741 (+ 733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:29.105746 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:29.105750 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:29.105912 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:29.106571 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:01:29.106575 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:01:29.106754 (+ 179us) client_negotiation.cc:769] Sending connection context
0807 07:01:29.106891 (+ 137us) client_negotiation.cc:240] Negotiation successful
0807 07:01:29.107188 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"thread_start_us":448,"threads_started":1}
I20260807 07:01:29.107604 23718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:29.090387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:46063 (local address 127.21.83.62:33131)
0807 07:01:29.090587 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:29.090594 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:29.091517 (+ 923us) server_negotiation.cc:407] Connection header received
0807 07:01:29.091775 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:29.091782 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:29.091880 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:29.092012 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:29.093201 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:29.094345 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:29.096322 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:29.097000 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:29.097159 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:29.099272 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:29.099279 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:29.099283 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:29.099327 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:29.104148 (+ 4821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:29.105247 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:29.105261 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:29.105264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:29.105484 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:29.106063 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:29.106068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:29.106072 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:29.106456 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:29.106638 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:29.107034 (+ 396us) server_negotiation.cc:299] Negotiation successful
0807 07:01:29.107287 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 07:01:29.107981 23672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
I20260807 07:01:29.110131 23903 heartbeater.cc:347] Connected to a master server at 127.21.83.62:33131
I20260807 07:01:29.110781 23903 heartbeater.cc:464] Registering TS with master...
I20260807 07:01:29.111547 23903 heartbeater.cc:511] Master 127.21.83.62:33131 requested a full tablet report, sending...
I20260807 07:01:29.113497 23565 ts_manager.cc:194] Registered new tserver with Master: 3d6807a83f544e32ae7874e242bf51ee (127.21.83.1:42307)
I20260807 07:01:29.113785 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 23672
I20260807 07:01:29.113970 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-0/wal/instance
I20260807 07:01:29.114590 21836 external_mini_cluster.cc:1428] Setting key 104f5ff203881e6880be21b06677afe8
I20260807 07:01:29.115018 23565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.1@KRBTEST.COM'} at 127.21.83.1:39343
WARNING: no policy specified for kudu/127.21.83.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab.
Entry for principal kudu/127.21.83.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab.
I20260807 07:01:29.160508 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.2:0
--local_ip_for_outbound_sockets=127.21.83.2
--webserver_interface=127.21.83.2
--webserver_port=0
--tserver_master_addrs=127.21.83.62:33131
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab
--principal=kudu/127.21.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:45239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:01:29.508244 23909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:01:29.508508 23909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:01:29.508607 23909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:01:29.515859 23909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:01:29.515981 23909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:01:29.516081 23909 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 07:01:29.516163 23909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.2
I20260807 07:01:29.526096 23909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:45239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab
--principal=kudu/127.21.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.83.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.83.62:33131
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.2
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:01:29.527577 23909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:01:29.529000 23909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:01:29.541769 23918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:29.541728 23921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:29.542382 23919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:01:29.542723 23909 server_base.cc:1034] running on GCE node
I20260807 07:01:29.543388 23909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:01:29.544319 23909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:01:29.545578 23909 hybrid_clock.cc:648] HybridClock initialized: now 1786086089545481 us; error 63 us; skew 500 ppm
I20260807 07:01:29.545897 23909 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:01:29 dist-test-slave-ncp9 krb5kdc[23502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086089, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:01:29.549103 23909 init.cc:377] Logged in from keytab as kudu/127.21.83.2@KRBTEST.COM (short username kudu)
I20260807 07:01:29.550532 23909 webserver.cc:460] Webserver started at http://127.21.83.2:44311/ using document root <none> and password file <none>
I20260807 07:01:29.551175 23909 fs_manager.cc:359] Metadata directory not provided
I20260807 07:01:29.551265 23909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:01:29.551548 23909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:01:29.554405 23909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-1/data/instance:
uuid: "a378a8686c4f4adc9bb13eda9783bb3d"
format_stamp: "Formatted at 2026-08-07 07:01:29 on dist-test-slave-ncp9"
server_key: "6d1bc22927e41c706bded06daece0d9c"
server_key_iv: "02117ba54a60b4038feddd775a0c3b4a"
server_key_version: "encryptionkey@0"
I20260807 07:01:29.555145 23909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-1/wal/instance:
uuid: "a378a8686c4f4adc9bb13eda9783bb3d"
format_stamp: "Formatted at 2026-08-07 07:01:29 on dist-test-slave-ncp9"
server_key: "6d1bc22927e41c706bded06daece0d9c"
server_key_iv: "02117ba54a60b4038feddd775a0c3b4a"
server_key_version: "encryptionkey@0"
I20260807 07:01:29.560410 23909 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260807 07:01:29.564334 23935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:29.565816 23909 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 07:01:29.565972 23909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-1/wal
uuid: "a378a8686c4f4adc9bb13eda9783bb3d"
format_stamp: "Formatted at 2026-08-07 07:01:29 on dist-test-slave-ncp9"
server_key: "6d1bc22927e41c706bded06daece0d9c"
server_key_iv: "02117ba54a60b4038feddd775a0c3b4a"
server_key_version: "encryptionkey@0"
I20260807 07:01:29.566196 23909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:01:29.581715 23909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:01:29.586627 23909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:01:29.586910 23909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:01:29.589879 23909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:01:29.589987 23909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:29.590139 23909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:01:29.590233 23909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 07:01:29 dist-test-slave-ncp9 krb5kdc[23502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086089, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.2@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:01:29.612567 23718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:29.590410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:34229 (local address 127.21.83.62:33131)
0807 07:01:29.590604 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:29.590611 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:29.592104 (+ 1493us) server_negotiation.cc:407] Connection header received
0807 07:01:29.594335 (+ 2231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:29.594340 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:29.594452 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:29.594542 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:29.596236 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:29.597234 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:29.600103 (+ 2869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:29.600659 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:29.600789 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:29.605352 (+ 4563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:29.605359 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:29.605365 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:29.605413 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:29.608472 (+ 3059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:29.609475 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:29.609481 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:29.609484 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:29.609653 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:29.610336 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:29.610339 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:29.610341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:29.610701 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:29.610841 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:29.611900 (+ 1059us) server_negotiation.cc:299] Negotiation successful
0807 07:01:29.612173 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260807 07:01:29.613657 23946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:29.591114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:33131 (local address 127.21.83.2:34229)
0807 07:01:29.591943 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:29.591986 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:29.594031 (+ 2045us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:29.594830 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:29.594842 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:29.595489 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:01:29.596069 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:29.596087 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:29.597429 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:29.597436 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:29.599940 (+ 2504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:29.599956 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:29.600917 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:29.600930 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:29.601163 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:29.601854 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:01:29.601888 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:01:29.605089 (+ 3201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:01:29.608619 (+ 3530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:29.608628 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:29.608647 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:29.609296 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:29.609777 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:29.609781 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:29.609783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:29.610192 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:29.610921 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:01:29.610928 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:01:29.611424 (+ 496us) client_negotiation.cc:769] Sending connection context
0807 07:01:29.611802 (+ 378us) client_negotiation.cc:240] Negotiation successful
0807 07:01:29.612227 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":373,"threads_started":1}
I20260807 07:01:29.633728 23718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:29.619470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:39799 (local address 127.21.83.62:33131)
0807 07:01:29.619652 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:29.619657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:29.620222 (+ 565us) server_negotiation.cc:407] Connection header received
0807 07:01:29.620386 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:29.620389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:29.620458 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:29.620537 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:29.621895 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:29.622830 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:29.624671 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:29.625230 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:29.625338 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:29.627679 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:29.627684 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:29.627687 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:29.627725 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:29.630758 (+ 3033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:29.631670 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:29.631675 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:29.631679 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:29.631851 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:29.632294 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:29.632298 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:29.632300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:29.632683 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:29.632797 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:29.633298 (+ 501us) server_negotiation.cc:299] Negotiation successful
0807 07:01:29.633499 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 07:01:29.634083 24043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:29.619457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:33131 (local address 127.21.83.2:39799)
0807 07:01:29.620114 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:29.620135 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:29.620282 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:29.620573 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:29.620578 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:29.621243 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:01:29.621740 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:29.621757 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:29.622961 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:29.622965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:29.624536 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:29.624552 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:29.625434 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:29.625440 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:29.625597 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:29.626316 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:01:29.626336 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:01:29.627415 (+ 1079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:01:29.630918 (+ 3503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:29.630923 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:29.630928 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:29.631514 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:29.632003 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:29.632009 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:29.632014 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:29.632186 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:29.632783 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:01:29.632788 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:01:29.633046 (+ 258us) client_negotiation.cc:769] Sending connection context
0807 07:01:29.633212 (+ 166us) client_negotiation.cc:240] Negotiation successful
0807 07:01:29.633573 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":348,"threads_started":1}
I20260807 07:01:29.641299 23909 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.2:40097
I20260807 07:01:29.641322 24111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.2:40097 every 8 connection(s)
I20260807 07:01:29.642477 23909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
I20260807 07:01:29.643409 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 23909
I20260807 07:01:29.643616 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-1/wal/instance
I20260807 07:01:29.644152 21836 external_mini_cluster.cc:1428] Setting key 4731e8030dce365a41f4fa4784e427b6
I20260807 07:01:29.644641 24112 heartbeater.cc:347] Connected to a master server at 127.21.83.62:33131
I20260807 07:01:29.644866 24112 heartbeater.cc:464] Registering TS with master...
I20260807 07:01:29.645529 24112 heartbeater.cc:511] Master 127.21.83.62:33131 requested a full tablet report, sending...
I20260807 07:01:29.646940 23564 ts_manager.cc:194] Registered new tserver with Master: a378a8686c4f4adc9bb13eda9783bb3d (127.21.83.2:40097)
I20260807 07:01:29.648067 23564 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.2@KRBTEST.COM'} at 127.21.83.2:34229
WARNING: no policy specified for kudu/127.21.83.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab.
Entry for principal kudu/127.21.83.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab.
I20260807 07:01:29.689092 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.3:0
--local_ip_for_outbound_sockets=127.21.83.3
--webserver_interface=127.21.83.3
--webserver_port=0
--tserver_master_addrs=127.21.83.62:33131
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab
--principal=kudu/127.21.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:45239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:01:30.018227 24116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:01:30.018503 24116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:01:30.018641 24116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:01:30.026180 24116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:01:30.026310 24116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:01:30.026438 24116 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 07:01:30.026546 24116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.3
I20260807 07:01:30.036886 24116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:45239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab
--principal=kudu/127.21.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.83.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.83.62:33131
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.3
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:01:30.038213 24116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:01:30.039490 24116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:01:30.051899 24129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:30.051966 24126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:30.051879 24127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:01:30.052847 24116 server_base.cc:1034] running on GCE node
I20260807 07:01:30.053428 24116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:01:30.054219 24116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:01:30.055421 24116 hybrid_clock.cc:648] HybridClock initialized: now 1786086090055363 us; error 29 us; skew 500 ppm
I20260807 07:01:30.055675 24116 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:01:30 dist-test-slave-ncp9 krb5kdc[23502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086090, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:01:30.058563 24116 init.cc:377] Logged in from keytab as kudu/127.21.83.3@KRBTEST.COM (short username kudu)
I20260807 07:01:30.059885 24116 webserver.cc:460] Webserver started at http://127.21.83.3:42453/ using document root <none> and password file <none>
I20260807 07:01:30.060489 24116 fs_manager.cc:359] Metadata directory not provided
I20260807 07:01:30.060601 24116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:01:30.060860 24116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:01:30.063807 24116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-2/data/instance:
uuid: "8b7aaeafa48249c7a5f29b048502bbe3"
format_stamp: "Formatted at 2026-08-07 07:01:30 on dist-test-slave-ncp9"
server_key: "6719f36c885eb36cda6c6ea58fa982c4"
server_key_iv: "1739c3406a749ad4056f136a10c70ef1"
server_key_version: "encryptionkey@0"
I20260807 07:01:30.064649 24116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-2/wal/instance:
uuid: "8b7aaeafa48249c7a5f29b048502bbe3"
format_stamp: "Formatted at 2026-08-07 07:01:30 on dist-test-slave-ncp9"
server_key: "6719f36c885eb36cda6c6ea58fa982c4"
server_key_iv: "1739c3406a749ad4056f136a10c70ef1"
server_key_version: "encryptionkey@0"
I20260807 07:01:30.070338 24116 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260807 07:01:30.074204 24142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:30.075807 24116 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 07:01:30.076000 24116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-2/wal
uuid: "8b7aaeafa48249c7a5f29b048502bbe3"
format_stamp: "Formatted at 2026-08-07 07:01:30 on dist-test-slave-ncp9"
server_key: "6719f36c885eb36cda6c6ea58fa982c4"
server_key_iv: "1739c3406a749ad4056f136a10c70ef1"
server_key_version: "encryptionkey@0"
I20260807 07:01:30.076258 24116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:01:30.096834 24116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:01:30.100703 24116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:01:30.100989 24116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:01:30.103580 24116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:01:30.103683 24116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:30.103830 24116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:01:30.103899 24116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:30.117866 23903 heartbeater.cc:503] Master 127.21.83.62:33131 was elected leader, sending a full tablet report...
Aug 07 07:01:30 dist-test-slave-ncp9 krb5kdc[23502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086090, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.3@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:01:30.128314 23718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:30.104366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:34213 (local address 127.21.83.62:33131)
0807 07:01:30.104549 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:30.104553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:30.106111 (+ 1558us) server_negotiation.cc:407] Connection header received
0807 07:01:30.109038 (+ 2927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:30.109044 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:30.109141 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:30.109209 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:30.111733 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:30.112777 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:30.115087 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:30.115652 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:30.115792 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:30.120925 (+ 5133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:30.120932 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:30.120937 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:30.120978 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:30.124120 (+ 3142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:30.125285 (+ 1165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:30.125291 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:30.125295 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:30.125469 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:30.125969 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:30.125976 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:30.125979 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:30.126398 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:30.126560 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:30.127610 (+ 1050us) server_negotiation.cc:299] Negotiation successful
0807 07:01:30.127872 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 07:01:30.129248 24153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:30.104879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:33131 (local address 127.21.83.3:34213)
0807 07:01:30.105929 (+ 1050us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:30.105977 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:30.108699 (+ 2722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:30.109730 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:30.109741 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:30.110663 (+ 922us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:01:30.111506 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:30.111527 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:30.112998 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:30.113004 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:30.114865 (+ 1861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:30.114885 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:30.115904 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:30.115912 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:30.116240 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:30.116979 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:01:30.117026 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:01:30.120605 (+ 3579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:01:30.124318 (+ 3713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:30.124328 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:30.124345 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:30.125137 (+ 792us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:30.125577 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:30.125582 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:30.125586 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:30.125826 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:30.126568 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:01:30.126577 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:01:30.127072 (+ 495us) client_negotiation.cc:769] Sending connection context
0807 07:01:30.127438 (+ 366us) client_negotiation.cc:240] Negotiation successful
0807 07:01:30.127932 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"thread_start_us":399,"threads_started":1}
I20260807 07:01:30.149596 24269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:30.134747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:33131 (local address 127.21.83.3:39731)
0807 07:01:30.135407 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:30.135429 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:30.135579 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:30.135898 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:30.135903 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:30.136653 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:01:30.137224 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:30.137243 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:30.138417 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:30.138423 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:30.140145 (+ 1722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:30.140162 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:30.141044 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:30.141049 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:30.141224 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:30.141986 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:01:30.142012 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:01:30.143124 (+ 1112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:01:30.146347 (+ 3223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:30.146353 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:30.146358 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:30.147043 (+ 685us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:30.147448 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:30.147453 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:30.147457 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:30.147637 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:30.148265 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:01:30.148274 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:01:30.148473 (+ 199us) client_negotiation.cc:769] Sending connection context
0807 07:01:30.148638 (+ 165us) client_negotiation.cc:240] Negotiation successful
0807 07:01:30.148950 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":344,"threads_started":1}
I20260807 07:01:30.149672 23718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:30.134775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:39731 (local address 127.21.83.62:33131)
0807 07:01:30.134966 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:30.134973 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:30.135513 (+ 540us) server_negotiation.cc:407] Connection header received
0807 07:01:30.135686 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:30.135690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:30.135766 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:30.135842 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:30.137368 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:30.138242 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:30.140297 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:30.140811 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:30.141611 (+ 800us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:30.143338 (+ 1727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:30.143345 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:30.143350 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:30.143390 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:30.146216 (+ 2826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:30.147181 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:30.147188 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:30.147192 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:30.147326 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:30.147806 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:30.147811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:30.147814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:30.148146 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:30.148281 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:30.149090 (+ 809us) server_negotiation.cc:299] Negotiation successful
0807 07:01:30.149321 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 07:01:30.157678 24116 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.3:40193
I20260807 07:01:30.157709 24329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.3:40193 every 8 connection(s)
I20260807 07:01:30.158838 24116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
I20260807 07:01:30.160718 24331 heartbeater.cc:347] Connected to a master server at 127.21.83.62:33131
I20260807 07:01:30.160933 24331 heartbeater.cc:464] Registering TS with master...
I20260807 07:01:30.161545 24331 heartbeater.cc:511] Master 127.21.83.62:33131 requested a full tablet report, sending...
I20260807 07:01:30.162784 23564 ts_manager.cc:194] Registered new tserver with Master: 8b7aaeafa48249c7a5f29b048502bbe3 (127.21.83.3:40193)
I20260807 07:01:30.163589 23564 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.3@KRBTEST.COM'} at 127.21.83.3:34213
I20260807 07:01:30.169291 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 24116
I20260807 07:01:30.169481 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-2/wal/instance
I20260807 07:01:30.169930 21836 external_mini_cluster.cc:1428] Setting key 4d33d946a2749946f046448fa583a8ee
I20260807 07:01:30.173130 21836 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 07:01:30 dist-test-slave-ncp9 krb5kdc[23502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086090, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 07:01:30 dist-test-slave-ncp9 krb5kdc[23502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086090, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:01:30.229378 23718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:30.212878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51136 (local address 127.21.83.62:33131)
0807 07:01:30.213129 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:30.213136 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:30.213281 (+ 145us) server_negotiation.cc:407] Connection header received
0807 07:01:30.213422 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:30.213427 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:30.213494 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:30.213587 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:30.214909 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:30.215716 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:30.217058 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:30.217548 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:30.217694 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:30.224127 (+ 6433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:30.224133 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:30.224136 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:30.224179 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:30.226782 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:30.227528 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:30.227533 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:30.227535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:30.227670 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:30.228035 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:30.228038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:30.228040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:30.228342 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:30.228549 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:30.228856 (+ 307us) server_negotiation.cc:299] Negotiation successful
0807 07:01:30.229078 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 07:01:30.233529 23564 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51136:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 07:01:30.240303 23564 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 07:01:30.274037 24358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:30.252683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39612 (local address 127.21.83.3:40193)
0807 07:01:30.265190 (+ 12507us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:30.265197 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:30.265231 (+ 34us) server_negotiation.cc:407] Connection header received
0807 07:01:30.265330 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:30.265337 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:30.265599 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:30.265799 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:01:30.266896 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:30.268438 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:30.271380 (+ 2942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:30.272817 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:30.272996 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:30.273263 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:30.273492 (+ 229us) server_negotiation.cc:299] Negotiation successful
0807 07:01:30.273672 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12338,"thread_start_us":12131,"threads_started":1}
I20260807 07:01:30.274200 24364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:30.252358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.3:40193 (local address 127.0.0.1:39612)
0807 07:01:30.262878 (+ 10520us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:30.262921 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:30.263114 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:30.265841 (+ 2727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:30.265850 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:30.265885 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:01:30.266469 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:30.266485 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:30.268692 (+ 2207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:30.268698 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:30.271227 (+ 2529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:30.271239 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:30.273055 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:30.273062 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:30.273265 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:30.273319 (+ 54us) client_negotiation.cc:769] Sending connection context
0807 07:01:30.273430 (+ 111us) client_negotiation.cc:240] Negotiation successful
0807 07:01:30.273717 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10232,"mutex_wait_us":144,"thread_start_us":16053,"threads_started":1}
I20260807 07:01:30.274698 24357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:30.252534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:42307 (local address 127.0.0.1:35192)
0807 07:01:30.264422 (+ 11888us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:30.264442 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:30.264562 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:30.265908 (+ 1346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:30.265913 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:30.265937 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:01:30.266548 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:30.266562 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:30.268408 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:30.268414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:30.271598 (+ 3184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:30.271612 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:30.273646 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:30.273652 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:30.273803 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:30.273836 (+ 33us) client_negotiation.cc:769] Sending connection context
0807 07:01:30.273933 (+ 97us) client_negotiation.cc:240] Negotiation successful
0807 07:01:30.274068 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11765,"mutex_wait_us":80,"thread_start_us":11608,"threads_started":1}
I20260807 07:01:30.274799 24360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:30.253036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35192 (local address 127.21.83.1:42307)
0807 07:01:30.265419 (+ 12383us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:30.265426 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:30.265449 (+ 23us) server_negotiation.cc:407] Connection header received
0807 07:01:30.265553 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:30.265560 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:30.265742 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:30.265919 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:01:30.266745 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:30.268268 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:30.271793 (+ 3525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:30.272829 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:30.273008 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:30.273240 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:30.274321 (+ 1081us) server_negotiation.cc:299] Negotiation successful
0807 07:01:30.274500 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12239,"thread_start_us":12100,"threads_started":1}
I20260807 07:01:30.277011 24353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:30.254218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.2:40097 (local address 127.0.0.1:56856)
0807 07:01:30.268717 (+ 14499us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:30.268733 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:30.268863 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:30.269456 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:30.269462 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:30.269498 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:01:30.270095 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:30.270112 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:30.271999 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:30.272004 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:30.274013 (+ 2009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:30.274033 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:30.276133 (+ 2100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:30.276140 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:30.276329 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:30.276362 (+ 33us) client_negotiation.cc:769] Sending connection context
0807 07:01:30.276457 (+ 95us) client_negotiation.cc:240] Negotiation successful
0807 07:01:30.276613 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14372,"mutex_wait_us":3,"thread_start_us":8252,"threads_started":1}
I20260807 07:01:30.277330 24366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:30.254434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56856 (local address 127.21.83.2:40097)
0807 07:01:30.261691 (+ 7257us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:30.261701 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:30.268840 (+ 7139us) server_negotiation.cc:407] Connection header received
0807 07:01:30.269028 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:30.269036 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:30.269261 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:30.269465 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:01:30.270303 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:30.271853 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:30.274233 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:30.275618 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:30.275748 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:30.275966 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:30.276532 (+ 566us) server_negotiation.cc:299] Negotiation successful
0807 07:01:30.276877 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7090,"thread_start_us":6961,"threads_started":1}
I20260807 07:01:30.278841 24217 tablet_service.cc:1467] Processing CreateTablet for tablet 90412c80f19a468fb453d37a9b37331b (DEFAULT_TABLE table=test-table [id=598cb982b77b4a7fb4f4bc307d3e5fd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:01:30.279304 23787 tablet_service.cc:1467] Processing CreateTablet for tablet 90412c80f19a468fb453d37a9b37331b (DEFAULT_TABLE table=test-table [id=598cb982b77b4a7fb4f4bc307d3e5fd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:01:30.280611 23787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90412c80f19a468fb453d37a9b37331b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:01:30.280681 24217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90412c80f19a468fb453d37a9b37331b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:01:30.280849 24013 tablet_service.cc:1467] Processing CreateTablet for tablet 90412c80f19a468fb453d37a9b37331b (DEFAULT_TABLE table=test-table [id=598cb982b77b4a7fb4f4bc307d3e5fd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:01:30.282481 24013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90412c80f19a468fb453d37a9b37331b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:01:30.293280 24367 tablet_bootstrap.cc:492] T 90412c80f19a468fb453d37a9b37331b P 3d6807a83f544e32ae7874e242bf51ee: Bootstrap starting.
I20260807 07:01:30.296186 24370 tablet_bootstrap.cc:492] T 90412c80f19a468fb453d37a9b37331b P 8b7aaeafa48249c7a5f29b048502bbe3: Bootstrap starting.
I20260807 07:01:30.296825 24367 tablet_bootstrap.cc:654] T 90412c80f19a468fb453d37a9b37331b P 3d6807a83f544e32ae7874e242bf51ee: Neither blocks nor log segments found. Creating new log.
I20260807 07:01:30.298195 24367 log.cc:824] T 90412c80f19a468fb453d37a9b37331b P 3d6807a83f544e32ae7874e242bf51ee: Log is configured to *not* fsync() on all Append() calls
I20260807 07:01:30.299772 24370 tablet_bootstrap.cc:654] T 90412c80f19a468fb453d37a9b37331b P 8b7aaeafa48249c7a5f29b048502bbe3: Neither blocks nor log segments found. Creating new log.
I20260807 07:01:30.301102 24370 log.cc:824] T 90412c80f19a468fb453d37a9b37331b P 8b7aaeafa48249c7a5f29b048502bbe3: Log is configured to *not* fsync() on all Append() calls
I20260807 07:01:30.302721 24371 tablet_bootstrap.cc:492] T 90412c80f19a468fb453d37a9b37331b P a378a8686c4f4adc9bb13eda9783bb3d: Bootstrap starting.
I20260807 07:01:30.304337 24370 tablet_bootstrap.cc:492] T 90412c80f19a468fb453d37a9b37331b P 8b7aaeafa48249c7a5f29b048502bbe3: No bootstrap required, opened a new log
I20260807 07:01:30.304626 24370 ts_tablet_manager.cc:1397] T 90412c80f19a468fb453d37a9b37331b P 8b7aaeafa48249c7a5f29b048502bbe3: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260807 07:01:30.305979 24367 tablet_bootstrap.cc:492] T 90412c80f19a468fb453d37a9b37331b P 3d6807a83f544e32ae7874e242bf51ee: No bootstrap required, opened a new log
I20260807 07:01:30.306010 24371 tablet_bootstrap.cc:654] T 90412c80f19a468fb453d37a9b37331b P a378a8686c4f4adc9bb13eda9783bb3d: Neither blocks nor log segments found. Creating new log.
I20260807 07:01:30.306236 24367 ts_tablet_manager.cc:1397] T 90412c80f19a468fb453d37a9b37331b P 3d6807a83f544e32ae7874e242bf51ee: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.006s
I20260807 07:01:30.307140 24371 log.cc:824] T 90412c80f19a468fb453d37a9b37331b P a378a8686c4f4adc9bb13eda9783bb3d: Log is configured to *not* fsync() on all Append() calls
I20260807 07:01:30.310250 24371 tablet_bootstrap.cc:492] T 90412c80f19a468fb453d37a9b37331b P a378a8686c4f4adc9bb13eda9783bb3d: No bootstrap required, opened a new log
I20260807 07:01:30.310655 24371 ts_tablet_manager.cc:1397] T 90412c80f19a468fb453d37a9b37331b P a378a8686c4f4adc9bb13eda9783bb3d: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260807 07:01:30.313645 24367 raft_consensus.cc:348] T 90412c80f19a468fb453d37a9b37331b P 3d6807a83f544e32ae7874e242bf51ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7aaeafa48249c7a5f29b048502bbe3" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40193 } } peers { permanent_uuid: "3d6807a83f544e32ae7874e242bf51ee" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42307 } } peers { permanent_uuid: "a378a8686c4f4adc9bb13eda9783bb3d" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40097 } }
I20260807 07:01:30.313966 24367 raft_consensus.cc:374] T 90412c80f19a468fb453d37a9b37331b P 3d6807a83f544e32ae7874e242bf51ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:01:30.314107 24367 raft_consensus.cc:729] T 90412c80f19a468fb453d37a9b37331b P 3d6807a83f544e32ae7874e242bf51ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d6807a83f544e32ae7874e242bf51ee, State: Initialized, Role: FOLLOWER
I20260807 07:01:30.314388 24370 raft_consensus.cc:348] T 90412c80f19a468fb453d37a9b37331b P 8b7aaeafa48249c7a5f29b048502bbe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7aaeafa48249c7a5f29b048502bbe3" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40193 } } peers { permanent_uuid: "3d6807a83f544e32ae7874e242bf51ee" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42307 } } peers { permanent_uuid: "a378a8686c4f4adc9bb13eda9783bb3d" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40097 } }
I20260807 07:01:30.314787 24370 raft_consensus.cc:374] T 90412c80f19a468fb453d37a9b37331b P 8b7aaeafa48249c7a5f29b048502bbe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:01:30.314644 24367 consensus_queue.cc:260] T 90412c80f19a468fb453d37a9b37331b P 3d6807a83f544e32ae7874e242bf51ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7aaeafa48249c7a5f29b048502bbe3" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40193 } } peers { permanent_uuid: "3d6807a83f544e32ae7874e242bf51ee" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42307 } } peers { permanent_uuid: "a378a8686c4f4adc9bb13eda9783bb3d" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40097 } }
I20260807 07:01:30.314884 24370 raft_consensus.cc:729] T 90412c80f19a468fb453d37a9b37331b P 8b7aaeafa48249c7a5f29b048502bbe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b7aaeafa48249c7a5f29b048502bbe3, State: Initialized, Role: FOLLOWER
I20260807 07:01:30.315516 24370 consensus_queue.cc:260] T 90412c80f19a468fb453d37a9b37331b P 8b7aaeafa48249c7a5f29b048502bbe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7aaeafa48249c7a5f29b048502bbe3" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40193 } } peers { permanent_uuid: "3d6807a83f544e32ae7874e242bf51ee" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42307 } } peers { permanent_uuid: "a378a8686c4f4adc9bb13eda9783bb3d" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40097 } }
I20260807 07:01:30.316362 24367 ts_tablet_manager.cc:1428] T 90412c80f19a468fb453d37a9b37331b P 3d6807a83f544e32ae7874e242bf51ee: Time spent starting tablet: real 0.010s user 0.007s sys 0.004s
I20260807 07:01:30.318050 24370 ts_tablet_manager.cc:1428] T 90412c80f19a468fb453d37a9b37331b P 8b7aaeafa48249c7a5f29b048502bbe3: Time spent starting tablet: real 0.013s user 0.008s sys 0.005s
I20260807 07:01:30.318490 24331 heartbeater.cc:503] Master 127.21.83.62:33131 was elected leader, sending a full tablet report...
I20260807 07:01:30.320242 24371 raft_consensus.cc:348] T 90412c80f19a468fb453d37a9b37331b P a378a8686c4f4adc9bb13eda9783bb3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7aaeafa48249c7a5f29b048502bbe3" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40193 } } peers { permanent_uuid: "3d6807a83f544e32ae7874e242bf51ee" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42307 } } peers { permanent_uuid: "a378a8686c4f4adc9bb13eda9783bb3d" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40097 } }
I20260807 07:01:30.320611 24371 raft_consensus.cc:374] T 90412c80f19a468fb453d37a9b37331b P a378a8686c4f4adc9bb13eda9783bb3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:01:30.320730 24371 raft_consensus.cc:729] T 90412c80f19a468fb453d37a9b37331b P a378a8686c4f4adc9bb13eda9783bb3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a378a8686c4f4adc9bb13eda9783bb3d, State: Initialized, Role: FOLLOWER
I20260807 07:01:30.321313 24371 consensus_queue.cc:260] T 90412c80f19a468fb453d37a9b37331b P a378a8686c4f4adc9bb13eda9783bb3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7aaeafa48249c7a5f29b048502bbe3" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40193 } } peers { permanent_uuid: "3d6807a83f544e32ae7874e242bf51ee" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42307 } } peers { permanent_uuid: "a378a8686c4f4adc9bb13eda9783bb3d" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40097 } }
I20260807 07:01:30.322325 24112 heartbeater.cc:503] Master 127.21.83.62:33131 was elected leader, sending a full tablet report...
I20260807 07:01:30.322950 24371 ts_tablet_manager.cc:1428] T 90412c80f19a468fb453d37a9b37331b P a378a8686c4f4adc9bb13eda9783bb3d: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
W20260807 07:01:30.360551 23904 tablet.cc:2369] T 90412c80f19a468fb453d37a9b37331b P 3d6807a83f544e32ae7874e242bf51ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 07:01:30.367041 24377 raft_consensus.cc:482] T 90412c80f19a468fb453d37a9b37331b P 3d6807a83f544e32ae7874e242bf51ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:01:30.367341 24377 raft_consensus.cc:504] T 90412c80f19a468fb453d37a9b37331b P 3d6807a83f544e32ae7874e242bf51ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7aaeafa48249c7a5f29b048502bbe3" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40193 } } peers { permanent_uuid: "3d6807a83f544e32ae7874e242bf51ee" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42307 } } peers { permanent_uuid: "a378a8686c4f4adc9bb13eda9783bb3d" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40097 } }
I20260807 07:01:30.369133 24377 leader_election.cc:288] T 90412c80f19a468fb453d37a9b37331b P 3d6807a83f544e32ae7874e242bf51ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b7aaeafa48249c7a5f29b048502bbe3 (127.21.83.3:40193), a378a8686c4f4adc9bb13eda9783bb3d (127.21.83.2:40097)
I20260807 07:01:30.376257 24358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:30.369676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:58501 (local address 127.21.83.3:40193)
0807 07:01:30.369877 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:30.369884 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:30.370785 (+ 901us) server_negotiation.cc:407] Connection header received
0807 07:01:30.370974 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:30.370980 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:30.371048 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:30.371122 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:01:30.371735 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:30.372473 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:30.374468 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:30.375272 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:30.375390 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:30.375511 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:30.375866 (+ 355us) server_negotiation.cc:299] Negotiation successful
0807 07:01:30.376005 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260807 07:01:30.376259 24366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:30.369610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:43753 (local address 127.21.83.2:40097)
0807 07:01:30.369837 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:30.369844 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:30.370830 (+ 986us) server_negotiation.cc:407] Connection header received
0807 07:01:30.371021 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:30.371026 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:30.371091 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:30.371173 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:01:30.371826 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:30.372554 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:30.374444 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:30.375254 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:30.375387 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:30.375510 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:30.375847 (+ 337us) server_negotiation.cc:299] Negotiation successful
0807 07:01:30.375987 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260807 07:01:30.376441 24380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:30.369683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.2:40097 (local address 127.21.83.1:43753)
0807 07:01:30.370719 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:30.370753 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:30.370890 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:30.371208 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:30.371212 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:30.371236 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:01:30.371677 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:30.371686 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:30.372690 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:30.372695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:30.374274 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:30.374292 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:30.375511 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:30.375521 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:30.375682 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:30.375711 (+ 29us) client_negotiation.cc:769] Sending connection context
0807 07:01:30.375801 (+ 90us) client_negotiation.cc:240] Negotiation successful
0807 07:01:30.375965 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":934,"mutex_wait_us":112,"thread_start_us":564,"threads_started":1}
I20260807 07:01:30.376793 24381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:30.369683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.3:40193 (local address 127.21.83.1:58501)
0807 07:01:30.370656 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:30.370676 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:30.370829 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:30.371158 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:30.371163 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:30.371187 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:01:30.371580 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:30.371594 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:30.372619 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:30.372624 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:30.374301 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:30.374315 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:30.375546 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:30.375553 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:30.375697 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:30.375726 (+ 29us) client_negotiation.cc:769] Sending connection context
0807 07:01:30.375808 (+ 82us) client_negotiation.cc:240] Negotiation successful
0807 07:01:30.375969 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":842,"mutex_wait_us":72,"thread_start_us":534,"threads_started":1}
I20260807 07:01:30.377640 24045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90412c80f19a468fb453d37a9b37331b" candidate_uuid: "3d6807a83f544e32ae7874e242bf51ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a378a8686c4f4adc9bb13eda9783bb3d" is_pre_election: true
I20260807 07:01:30.377743 24251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90412c80f19a468fb453d37a9b37331b" candidate_uuid: "3d6807a83f544e32ae7874e242bf51ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b7aaeafa48249c7a5f29b048502bbe3" is_pre_election: true
I20260807 07:01:30.378291 24251 raft_consensus.cc:2454] T 90412c80f19a468fb453d37a9b37331b P 8b7aaeafa48249c7a5f29b048502bbe3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d6807a83f544e32ae7874e242bf51ee in term 0.
I20260807 07:01:30.378291 24045 raft_consensus.cc:2454] T 90412c80f19a468fb453d37a9b37331b P a378a8686c4f4adc9bb13eda9783bb3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d6807a83f544e32ae7874e242bf51ee in term 0.
I20260807 07:01:30.379136 23705 leader_election.cc:302] T 90412c80f19a468fb453d37a9b37331b P 3d6807a83f544e32ae7874e242bf51ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d6807a83f544e32ae7874e242bf51ee, a378a8686c4f4adc9bb13eda9783bb3d; no voters:
I20260807 07:01:30.379637 24377 raft_consensus.cc:2781] T 90412c80f19a468fb453d37a9b37331b P 3d6807a83f544e32ae7874e242bf51ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 07:01:30.379794 24377 raft_consensus.cc:482] T 90412c80f19a468fb453d37a9b37331b P 3d6807a83f544e32ae7874e242bf51ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:01:30.379894 24377 raft_consensus.cc:3037] T 90412c80f19a468fb453d37a9b37331b P 3d6807a83f544e32ae7874e242bf51ee [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:01:30.382510 24377 raft_consensus.cc:504] T 90412c80f19a468fb453d37a9b37331b P 3d6807a83f544e32ae7874e242bf51ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7aaeafa48249c7a5f29b048502bbe3" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40193 } } peers { permanent_uuid: "3d6807a83f544e32ae7874e242bf51ee" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42307 } } peers { permanent_uuid: "a378a8686c4f4adc9bb13eda9783bb3d" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40097 } }
I20260807 07:01:30.383347 24377 leader_election.cc:288] T 90412c80f19a468fb453d37a9b37331b P 3d6807a83f544e32ae7874e242bf51ee [CANDIDATE]: Term 1 election: Requested vote from peers 8b7aaeafa48249c7a5f29b048502bbe3 (127.21.83.3:40193), a378a8686c4f4adc9bb13eda9783bb3d (127.21.83.2:40097)
I20260807 07:01:30.383792 24251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90412c80f19a468fb453d37a9b37331b" candidate_uuid: "3d6807a83f544e32ae7874e242bf51ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b7aaeafa48249c7a5f29b048502bbe3"
I20260807 07:01:30.384044 24251 raft_consensus.cc:3037] T 90412c80f19a468fb453d37a9b37331b P 8b7aaeafa48249c7a5f29b048502bbe3 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:01:30.384020 24045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90412c80f19a468fb453d37a9b37331b" candidate_uuid: "3d6807a83f544e32ae7874e242bf51ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a378a8686c4f4adc9bb13eda9783bb3d"
I20260807 07:01:30.384259 24045 raft_consensus.cc:3037] T 90412c80f19a468fb453d37a9b37331b P a378a8686c4f4adc9bb13eda9783bb3d [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:01:30.386699 24045 raft_consensus.cc:2454] T 90412c80f19a468fb453d37a9b37331b P a378a8686c4f4adc9bb13eda9783bb3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d6807a83f544e32ae7874e242bf51ee in term 1.
I20260807 07:01:30.386709 24251 raft_consensus.cc:2454] T 90412c80f19a468fb453d37a9b37331b P 8b7aaeafa48249c7a5f29b048502bbe3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d6807a83f544e32ae7874e242bf51ee in term 1.
I20260807 07:01:30.387169 23705 leader_election.cc:302] T 90412c80f19a468fb453d37a9b37331b P 3d6807a83f544e32ae7874e242bf51ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d6807a83f544e32ae7874e242bf51ee, a378a8686c4f4adc9bb13eda9783bb3d; no voters:
I20260807 07:01:30.387543 24377 raft_consensus.cc:2781] T 90412c80f19a468fb453d37a9b37331b P 3d6807a83f544e32ae7874e242bf51ee [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:01:30.387998 24377 raft_consensus.cc:686] T 90412c80f19a468fb453d37a9b37331b P 3d6807a83f544e32ae7874e242bf51ee [term 1 LEADER]: Becoming Leader. State: Replica: 3d6807a83f544e32ae7874e242bf51ee, State: Running, Role: LEADER
I20260807 07:01:30.388543 24377 consensus_queue.cc:237] T 90412c80f19a468fb453d37a9b37331b P 3d6807a83f544e32ae7874e242bf51ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7aaeafa48249c7a5f29b048502bbe3" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40193 } } peers { permanent_uuid: "3d6807a83f544e32ae7874e242bf51ee" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42307 } } peers { permanent_uuid: "a378a8686c4f4adc9bb13eda9783bb3d" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40097 } }
I20260807 07:01:30.393380 23564 catalog_manager.cc:5495] T 90412c80f19a468fb453d37a9b37331b P 3d6807a83f544e32ae7874e242bf51ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d6807a83f544e32ae7874e242bf51ee (127.21.83.1). New cstate: current_term: 1 leader_uuid: "3d6807a83f544e32ae7874e242bf51ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7aaeafa48249c7a5f29b048502bbe3" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d6807a83f544e32ae7874e242bf51ee" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a378a8686c4f4adc9bb13eda9783bb3d" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40097 } health_report { overall_health: UNKNOWN } } }
W20260807 07:01:30.394793 24113 tablet.cc:2369] T 90412c80f19a468fb453d37a9b37331b P a378a8686c4f4adc9bb13eda9783bb3d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 07:01:30.410317 24332 tablet.cc:2369] T 90412c80f19a468fb453d37a9b37331b P 8b7aaeafa48249c7a5f29b048502bbe3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 07:01:30.430706 24360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:30.425284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35198 (local address 127.21.83.1:42307)
0807 07:01:30.425446 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:30.425450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:30.425497 (+ 47us) server_negotiation.cc:407] Connection header received
0807 07:01:30.425796 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:30.425800 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:30.425889 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:30.426006 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:01:30.426384 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:30.427091 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:30.428495 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:30.428971 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:30.429064 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:30.429617 (+ 553us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:01:30.430156 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:01:30.430267 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:30.430396 (+ 129us) server_negotiation.cc:299] Negotiation successful
0807 07:01:30.430469 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260807 07:01:30.438784 24045 raft_consensus.cc:1261] T 90412c80f19a468fb453d37a9b37331b P a378a8686c4f4adc9bb13eda9783bb3d [term 1 FOLLOWER]: Refusing update from remote peer 3d6807a83f544e32ae7874e242bf51ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:01:30.438782 24251 raft_consensus.cc:1261] T 90412c80f19a468fb453d37a9b37331b P 8b7aaeafa48249c7a5f29b048502bbe3 [term 1 FOLLOWER]: Refusing update from remote peer 3d6807a83f544e32ae7874e242bf51ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:01:30.439782 24383 consensus_queue.cc:1035] T 90412c80f19a468fb453d37a9b37331b P 3d6807a83f544e32ae7874e242bf51ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "a378a8686c4f4adc9bb13eda9783bb3d" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:01:30.440151 24377 consensus_queue.cc:1035] T 90412c80f19a468fb453d37a9b37331b P 3d6807a83f544e32ae7874e242bf51ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b7aaeafa48249c7a5f29b048502bbe3" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:01:30.475806 23565 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:01:30.476050 23565 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:01:30.479853 23565 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e5bf836233d747769243d412d635e83f: Sending DeleteTablet for 3 replicas of tablet 90412c80f19a468fb453d37a9b37331b
I20260807 07:01:30.481088 24013 tablet_service.cc:1514] Processing DeleteTablet for tablet 90412c80f19a468fb453d37a9b37331b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:01:30 UTC) from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:56856
I20260807 07:01:30.481093 23787 tablet_service.cc:1514] Processing DeleteTablet for tablet 90412c80f19a468fb453d37a9b37331b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:01:30 UTC) from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:35192
I20260807 07:01:30.481158 24217 tablet_service.cc:1514] Processing DeleteTablet for tablet 90412c80f19a468fb453d37a9b37331b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:01:30 UTC) from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:39612
I20260807 07:01:30.486897 23564 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51136:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 07:01:30.487613 23564 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 07:01:30.490638 24403 tablet_replica.cc:330] T 90412c80f19a468fb453d37a9b37331b P 8b7aaeafa48249c7a5f29b048502bbe3: stopping tablet replica
I20260807 07:01:30.491320 24403 raft_consensus.cc:2229] T 90412c80f19a468fb453d37a9b37331b P 8b7aaeafa48249c7a5f29b048502bbe3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:01:30.491891 24403 raft_consensus.cc:2258] T 90412c80f19a468fb453d37a9b37331b P 8b7aaeafa48249c7a5f29b048502bbe3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:01:30.492511 24401 tablet_replica.cc:330] T 90412c80f19a468fb453d37a9b37331b P a378a8686c4f4adc9bb13eda9783bb3d: stopping tablet replica
I20260807 07:01:30.493726 24401 raft_consensus.cc:2229] T 90412c80f19a468fb453d37a9b37331b P a378a8686c4f4adc9bb13eda9783bb3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:01:30.495385 24403 ts_tablet_manager.cc:1905] T 90412c80f19a468fb453d37a9b37331b P 8b7aaeafa48249c7a5f29b048502bbe3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:01:30.495997 24401 raft_consensus.cc:2258] T 90412c80f19a468fb453d37a9b37331b P a378a8686c4f4adc9bb13eda9783bb3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:01:30.497174 23786 tablet_service.cc:1467] Processing CreateTablet for tablet 7dee679534a045da9f0c97e014a15065 (DEFAULT_TABLE table=test-table [id=70f71a645e9e444bbb50c1cee0ebf978]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:01:30.497786 23786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dee679534a045da9f0c97e014a15065. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:01:30.501794 24217 tablet_service.cc:1467] Processing CreateTablet for tablet 7dee679534a045da9f0c97e014a15065 (DEFAULT_TABLE table=test-table [id=70f71a645e9e444bbb50c1cee0ebf978]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:01:30.502113 24011 tablet_service.cc:1467] Processing CreateTablet for tablet 7dee679534a045da9f0c97e014a15065 (DEFAULT_TABLE table=test-table [id=70f71a645e9e444bbb50c1cee0ebf978]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:01:30.502343 24217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dee679534a045da9f0c97e014a15065. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:01:30.502619 24011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dee679534a045da9f0c97e014a15065. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:01:30.508040 24403 ts_tablet_manager.cc:1918] T 90412c80f19a468fb453d37a9b37331b P 8b7aaeafa48249c7a5f29b048502bbe3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:01:30.508814 24402 tablet_replica.cc:330] T 90412c80f19a468fb453d37a9b37331b P 3d6807a83f544e32ae7874e242bf51ee: stopping tablet replica
I20260807 07:01:30.509380 24402 raft_consensus.cc:2229] T 90412c80f19a468fb453d37a9b37331b P 3d6807a83f544e32ae7874e242bf51ee [term 1 LEADER]: Raft consensus shutting down.
I20260807 07:01:30.510197 24401 ts_tablet_manager.cc:1905] T 90412c80f19a468fb453d37a9b37331b P a378a8686c4f4adc9bb13eda9783bb3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:01:30.510329 24402 raft_consensus.cc:2258] T 90412c80f19a468fb453d37a9b37331b P 3d6807a83f544e32ae7874e242bf51ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:01:30.512267 24402 ts_tablet_manager.cc:1905] T 90412c80f19a468fb453d37a9b37331b P 3d6807a83f544e32ae7874e242bf51ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:01:30.513485 24403 log.cc:1196] T 90412c80f19a468fb453d37a9b37331b P 8b7aaeafa48249c7a5f29b048502bbe3: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-2/wal/wals/90412c80f19a468fb453d37a9b37331b
I20260807 07:01:30.513902 24370 tablet_bootstrap.cc:492] T 7dee679534a045da9f0c97e014a15065 P 8b7aaeafa48249c7a5f29b048502bbe3: Bootstrap starting.
I20260807 07:01:30.515939 24371 tablet_bootstrap.cc:492] T 7dee679534a045da9f0c97e014a15065 P a378a8686c4f4adc9bb13eda9783bb3d: Bootstrap starting.
I20260807 07:01:30.517190 24370 tablet_bootstrap.cc:654] T 7dee679534a045da9f0c97e014a15065 P 8b7aaeafa48249c7a5f29b048502bbe3: Neither blocks nor log segments found. Creating new log.
I20260807 07:01:30.517710 24371 tablet_bootstrap.cc:654] T 7dee679534a045da9f0c97e014a15065 P a378a8686c4f4adc9bb13eda9783bb3d: Neither blocks nor log segments found. Creating new log.
I20260807 07:01:30.518615 24401 ts_tablet_manager.cc:1918] T 90412c80f19a468fb453d37a9b37331b P a378a8686c4f4adc9bb13eda9783bb3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:01:30.518805 24401 log.cc:1196] T 90412c80f19a468fb453d37a9b37331b P a378a8686c4f4adc9bb13eda9783bb3d: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-1/wal/wals/90412c80f19a468fb453d37a9b37331b
I20260807 07:01:30.519397 24401 ts_tablet_manager.cc:1939] T 90412c80f19a468fb453d37a9b37331b P a378a8686c4f4adc9bb13eda9783bb3d: Deleting consensus metadata
I20260807 07:01:30.521315 23543 catalog_manager.cc:4841] TS a378a8686c4f4adc9bb13eda9783bb3d (127.21.83.2:40097): tablet 90412c80f19a468fb453d37a9b37331b (table test-table [id=598cb982b77b4a7fb4f4bc307d3e5fd7]) successfully deleted
I20260807 07:01:30.522181 24403 ts_tablet_manager.cc:1939] T 90412c80f19a468fb453d37a9b37331b P 8b7aaeafa48249c7a5f29b048502bbe3: Deleting consensus metadata
I20260807 07:01:30.523371 24367 tablet_bootstrap.cc:492] T 7dee679534a045da9f0c97e014a15065 P 3d6807a83f544e32ae7874e242bf51ee: Bootstrap starting.
I20260807 07:01:30.525175 24367 tablet_bootstrap.cc:654] T 7dee679534a045da9f0c97e014a15065 P 3d6807a83f544e32ae7874e242bf51ee: Neither blocks nor log segments found. Creating new log.
I20260807 07:01:30.528730 24370 tablet_bootstrap.cc:492] T 7dee679534a045da9f0c97e014a15065 P 8b7aaeafa48249c7a5f29b048502bbe3: No bootstrap required, opened a new log
I20260807 07:01:30.530341 24370 ts_tablet_manager.cc:1397] T 7dee679534a045da9f0c97e014a15065 P 8b7aaeafa48249c7a5f29b048502bbe3: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.005s
I20260807 07:01:30.531281 24370 raft_consensus.cc:348] T 7dee679534a045da9f0c97e014a15065 P 8b7aaeafa48249c7a5f29b048502bbe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6807a83f544e32ae7874e242bf51ee" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42307 } } peers { permanent_uuid: "a378a8686c4f4adc9bb13eda9783bb3d" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40097 } } peers { permanent_uuid: "8b7aaeafa48249c7a5f29b048502bbe3" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40193 } }
I20260807 07:01:30.531607 24370 raft_consensus.cc:374] T 7dee679534a045da9f0c97e014a15065 P 8b7aaeafa48249c7a5f29b048502bbe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:01:30.531754 24370 raft_consensus.cc:729] T 7dee679534a045da9f0c97e014a15065 P 8b7aaeafa48249c7a5f29b048502bbe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b7aaeafa48249c7a5f29b048502bbe3, State: Initialized, Role: FOLLOWER
I20260807 07:01:30.532141 24370 consensus_queue.cc:260] T 7dee679534a045da9f0c97e014a15065 P 8b7aaeafa48249c7a5f29b048502bbe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6807a83f544e32ae7874e242bf51ee" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42307 } } peers { permanent_uuid: "a378a8686c4f4adc9bb13eda9783bb3d" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40097 } } peers { permanent_uuid: "8b7aaeafa48249c7a5f29b048502bbe3" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40193 } }
I20260807 07:01:30.532847 24370 ts_tablet_manager.cc:1428] T 7dee679534a045da9f0c97e014a15065 P 8b7aaeafa48249c7a5f29b048502bbe3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260807 07:01:30.535169 23546 catalog_manager.cc:4841] TS 8b7aaeafa48249c7a5f29b048502bbe3 (127.21.83.3:40193): tablet 90412c80f19a468fb453d37a9b37331b (table test-table [id=598cb982b77b4a7fb4f4bc307d3e5fd7]) successfully deleted
I20260807 07:01:30.542868 24367 tablet_bootstrap.cc:492] T 7dee679534a045da9f0c97e014a15065 P 3d6807a83f544e32ae7874e242bf51ee: No bootstrap required, opened a new log
I20260807 07:01:30.543118 24367 ts_tablet_manager.cc:1397] T 7dee679534a045da9f0c97e014a15065 P 3d6807a83f544e32ae7874e242bf51ee: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.008s
I20260807 07:01:30.543762 24402 ts_tablet_manager.cc:1918] T 90412c80f19a468fb453d37a9b37331b P 3d6807a83f544e32ae7874e242bf51ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:01:30.544034 24371 tablet_bootstrap.cc:492] T 7dee679534a045da9f0c97e014a15065 P a378a8686c4f4adc9bb13eda9783bb3d: No bootstrap required, opened a new log
I20260807 07:01:30.544044 24402 log.cc:1196] T 90412c80f19a468fb453d37a9b37331b P 3d6807a83f544e32ae7874e242bf51ee: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-0/wal/wals/90412c80f19a468fb453d37a9b37331b
I20260807 07:01:30.544241 24371 ts_tablet_manager.cc:1397] T 7dee679534a045da9f0c97e014a15065 P a378a8686c4f4adc9bb13eda9783bb3d: Time spent bootstrapping tablet: real 0.028s user 0.013s sys 0.007s
I20260807 07:01:30.544231 24367 raft_consensus.cc:348] T 7dee679534a045da9f0c97e014a15065 P 3d6807a83f544e32ae7874e242bf51ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6807a83f544e32ae7874e242bf51ee" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42307 } } peers { permanent_uuid: "a378a8686c4f4adc9bb13eda9783bb3d" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40097 } } peers { permanent_uuid: "8b7aaeafa48249c7a5f29b048502bbe3" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40193 } }
I20260807 07:01:30.544581 24367 raft_consensus.cc:374] T 7dee679534a045da9f0c97e014a15065 P 3d6807a83f544e32ae7874e242bf51ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:01:30.544750 24367 raft_consensus.cc:729] T 7dee679534a045da9f0c97e014a15065 P 3d6807a83f544e32ae7874e242bf51ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d6807a83f544e32ae7874e242bf51ee, State: Initialized, Role: FOLLOWER
I20260807 07:01:30.544785 24402 ts_tablet_manager.cc:1939] T 90412c80f19a468fb453d37a9b37331b P 3d6807a83f544e32ae7874e242bf51ee: Deleting consensus metadata
I20260807 07:01:30.545181 24371 raft_consensus.cc:348] T 7dee679534a045da9f0c97e014a15065 P a378a8686c4f4adc9bb13eda9783bb3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6807a83f544e32ae7874e242bf51ee" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42307 } } peers { permanent_uuid: "a378a8686c4f4adc9bb13eda9783bb3d" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40097 } } peers { permanent_uuid: "8b7aaeafa48249c7a5f29b048502bbe3" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40193 } }
I20260807 07:01:30.545200 24367 consensus_queue.cc:260] T 7dee679534a045da9f0c97e014a15065 P 3d6807a83f544e32ae7874e242bf51ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6807a83f544e32ae7874e242bf51ee" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42307 } } peers { permanent_uuid: "a378a8686c4f4adc9bb13eda9783bb3d" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40097 } } peers { permanent_uuid: "8b7aaeafa48249c7a5f29b048502bbe3" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40193 } }
I20260807 07:01:30.545444 24371 raft_consensus.cc:374] T 7dee679534a045da9f0c97e014a15065 P a378a8686c4f4adc9bb13eda9783bb3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:01:30.545584 24371 raft_consensus.cc:729] T 7dee679534a045da9f0c97e014a15065 P a378a8686c4f4adc9bb13eda9783bb3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a378a8686c4f4adc9bb13eda9783bb3d, State: Initialized, Role: FOLLOWER
I20260807 07:01:30.545862 24371 consensus_queue.cc:260] T 7dee679534a045da9f0c97e014a15065 P a378a8686c4f4adc9bb13eda9783bb3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6807a83f544e32ae7874e242bf51ee" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42307 } } peers { permanent_uuid: "a378a8686c4f4adc9bb13eda9783bb3d" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40097 } } peers { permanent_uuid: "8b7aaeafa48249c7a5f29b048502bbe3" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40193 } }
I20260807 07:01:30.546893 23545 catalog_manager.cc:4841] TS 3d6807a83f544e32ae7874e242bf51ee (127.21.83.1:42307): tablet 90412c80f19a468fb453d37a9b37331b (table test-table [id=598cb982b77b4a7fb4f4bc307d3e5fd7]) successfully deleted
I20260807 07:01:30.548286 24367 ts_tablet_manager.cc:1428] T 7dee679534a045da9f0c97e014a15065 P 3d6807a83f544e32ae7874e242bf51ee: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260807 07:01:30.548538 24371 ts_tablet_manager.cc:1428] T 7dee679534a045da9f0c97e014a15065 P a378a8686c4f4adc9bb13eda9783bb3d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260807 07:01:30.988392 24377 raft_consensus.cc:482] T 7dee679534a045da9f0c97e014a15065 P 3d6807a83f544e32ae7874e242bf51ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:01:30.988607 24377 raft_consensus.cc:504] T 7dee679534a045da9f0c97e014a15065 P 3d6807a83f544e32ae7874e242bf51ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6807a83f544e32ae7874e242bf51ee" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42307 } } peers { permanent_uuid: "a378a8686c4f4adc9bb13eda9783bb3d" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40097 } } peers { permanent_uuid: "8b7aaeafa48249c7a5f29b048502bbe3" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40193 } }
I20260807 07:01:30.989259 24377 leader_election.cc:288] T 7dee679534a045da9f0c97e014a15065 P 3d6807a83f544e32ae7874e242bf51ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a378a8686c4f4adc9bb13eda9783bb3d (127.21.83.2:40097), 8b7aaeafa48249c7a5f29b048502bbe3 (127.21.83.3:40193)
I20260807 07:01:30.989923 24251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dee679534a045da9f0c97e014a15065" candidate_uuid: "3d6807a83f544e32ae7874e242bf51ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b7aaeafa48249c7a5f29b048502bbe3" is_pre_election: true
I20260807 07:01:30.989984 24045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dee679534a045da9f0c97e014a15065" candidate_uuid: "3d6807a83f544e32ae7874e242bf51ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a378a8686c4f4adc9bb13eda9783bb3d" is_pre_election: true
I20260807 07:01:30.990154 24251 raft_consensus.cc:2454] T 7dee679534a045da9f0c97e014a15065 P 8b7aaeafa48249c7a5f29b048502bbe3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d6807a83f544e32ae7874e242bf51ee in term 0.
I20260807 07:01:30.990275 24045 raft_consensus.cc:2454] T 7dee679534a045da9f0c97e014a15065 P a378a8686c4f4adc9bb13eda9783bb3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d6807a83f544e32ae7874e242bf51ee in term 0.
I20260807 07:01:30.990638 23711 leader_election.cc:302] T 7dee679534a045da9f0c97e014a15065 P 3d6807a83f544e32ae7874e242bf51ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d6807a83f544e32ae7874e242bf51ee, 8b7aaeafa48249c7a5f29b048502bbe3; no voters:
I20260807 07:01:30.990942 24377 raft_consensus.cc:2781] T 7dee679534a045da9f0c97e014a15065 P 3d6807a83f544e32ae7874e242bf51ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 07:01:30.991062 24377 raft_consensus.cc:482] T 7dee679534a045da9f0c97e014a15065 P 3d6807a83f544e32ae7874e242bf51ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:01:30.991158 24377 raft_consensus.cc:3037] T 7dee679534a045da9f0c97e014a15065 P 3d6807a83f544e32ae7874e242bf51ee [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:01:30.993558 24377 raft_consensus.cc:504] T 7dee679534a045da9f0c97e014a15065 P 3d6807a83f544e32ae7874e242bf51ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6807a83f544e32ae7874e242bf51ee" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42307 } } peers { permanent_uuid: "a378a8686c4f4adc9bb13eda9783bb3d" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40097 } } peers { permanent_uuid: "8b7aaeafa48249c7a5f29b048502bbe3" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40193 } }
I20260807 07:01:30.994207 24377 leader_election.cc:288] T 7dee679534a045da9f0c97e014a15065 P 3d6807a83f544e32ae7874e242bf51ee [CANDIDATE]: Term 1 election: Requested vote from peers a378a8686c4f4adc9bb13eda9783bb3d (127.21.83.2:40097), 8b7aaeafa48249c7a5f29b048502bbe3 (127.21.83.3:40193)
I20260807 07:01:30.994803 24045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dee679534a045da9f0c97e014a15065" candidate_uuid: "3d6807a83f544e32ae7874e242bf51ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a378a8686c4f4adc9bb13eda9783bb3d"
I20260807 07:01:30.994839 24251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dee679534a045da9f0c97e014a15065" candidate_uuid: "3d6807a83f544e32ae7874e242bf51ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b7aaeafa48249c7a5f29b048502bbe3"
I20260807 07:01:30.995056 24251 raft_consensus.cc:3037] T 7dee679534a045da9f0c97e014a15065 P 8b7aaeafa48249c7a5f29b048502bbe3 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:01:30.995061 24045 raft_consensus.cc:3037] T 7dee679534a045da9f0c97e014a15065 P a378a8686c4f4adc9bb13eda9783bb3d [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:01:30.997972 24251 raft_consensus.cc:2454] T 7dee679534a045da9f0c97e014a15065 P 8b7aaeafa48249c7a5f29b048502bbe3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d6807a83f544e32ae7874e242bf51ee in term 1.
I20260807 07:01:30.997972 24045 raft_consensus.cc:2454] T 7dee679534a045da9f0c97e014a15065 P a378a8686c4f4adc9bb13eda9783bb3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d6807a83f544e32ae7874e242bf51ee in term 1.
I20260807 07:01:30.998515 23705 leader_election.cc:302] T 7dee679534a045da9f0c97e014a15065 P 3d6807a83f544e32ae7874e242bf51ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3d6807a83f544e32ae7874e242bf51ee, 8b7aaeafa48249c7a5f29b048502bbe3, a378a8686c4f4adc9bb13eda9783bb3d; no voters:
I20260807 07:01:30.998852 24377 raft_consensus.cc:2781] T 7dee679534a045da9f0c97e014a15065 P 3d6807a83f544e32ae7874e242bf51ee [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:01:30.999223 24377 raft_consensus.cc:686] T 7dee679534a045da9f0c97e014a15065 P 3d6807a83f544e32ae7874e242bf51ee [term 1 LEADER]: Becoming Leader. State: Replica: 3d6807a83f544e32ae7874e242bf51ee, State: Running, Role: LEADER
I20260807 07:01:30.999486 24377 consensus_queue.cc:237] T 7dee679534a045da9f0c97e014a15065 P 3d6807a83f544e32ae7874e242bf51ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6807a83f544e32ae7874e242bf51ee" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42307 } } peers { permanent_uuid: "a378a8686c4f4adc9bb13eda9783bb3d" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40097 } } peers { permanent_uuid: "8b7aaeafa48249c7a5f29b048502bbe3" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40193 } }
I20260807 07:01:31.002038 23564 catalog_manager.cc:5495] T 7dee679534a045da9f0c97e014a15065 P 3d6807a83f544e32ae7874e242bf51ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d6807a83f544e32ae7874e242bf51ee (127.21.83.1). New cstate: current_term: 1 leader_uuid: "3d6807a83f544e32ae7874e242bf51ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6807a83f544e32ae7874e242bf51ee" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a378a8686c4f4adc9bb13eda9783bb3d" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b7aaeafa48249c7a5f29b048502bbe3" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40193 } health_report { overall_health: UNKNOWN } } }
Aug 07 07:01:31 dist-test-slave-ncp9 krb5kdc[23502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086088, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:01:31.129108 24425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:31.115212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:33131 (local address 127.0.0.1:51150)
0807 07:01:31.116068 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:31.116085 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:31.116202 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:31.116540 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:31.116543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:31.117238 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:01:31.117431 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:31.117440 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:31.118870 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:31.118873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:31.119789 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:31.119801 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:31.120663 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:31.120667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:31.120796 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:31.121469 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:01:31.121486 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:01:31.123997 (+ 2511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:01:31.126902 (+ 2905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:31.126912 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:31.126918 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:31.127387 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:31.127705 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:31.127708 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:31.127710 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:31.127844 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:31.128267 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:01:31.128275 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:01:31.128434 (+ 159us) client_negotiation.cc:769] Sending connection context
0807 07:01:31.128580 (+ 146us) client_negotiation.cc:240] Negotiation successful
0807 07:01:31.128791 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"thread_start_us":576,"threads_started":1}
I20260807 07:01:31.129261 24424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:31.115204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51150 (local address 127.21.83.62:33131)
0807 07:01:31.115877 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:31.115883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:31.116161 (+ 278us) server_negotiation.cc:407] Connection header received
0807 07:01:31.116307 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:31.116312 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:31.116389 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:31.116506 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:31.117566 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:31.118731 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:31.119920 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:31.120521 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:31.120653 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:31.124167 (+ 3514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:31.124171 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:31.124175 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:31.124203 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:31.126800 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:31.127477 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:31.127480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:31.127483 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:31.127613 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:31.127925 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:31.127928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:31.127930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:31.128168 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:31.128322 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:31.128681 (+ 359us) server_negotiation.cc:299] Negotiation successful
0807 07:01:31.128922 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":497,"thread_start_us":343,"threads_started":1}
I20260807 07:01:31.133884 23564 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:51150:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260807 07:01:31.134799 23564 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 07:01:31.142845 24011 tablet_service.cc:1467] Processing CreateTablet for tablet 0f2a4973d62c40df9ded5798c13452a6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d0f3ec9e77f34d0ca1399ecb757d7430]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 07:01:31.142817 23786 tablet_service.cc:1467] Processing CreateTablet for tablet 0f2a4973d62c40df9ded5798c13452a6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d0f3ec9e77f34d0ca1399ecb757d7430]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 07:01:31.143489 23786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f2a4973d62c40df9ded5798c13452a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:01:31.143487 24011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f2a4973d62c40df9ded5798c13452a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:01:31.143290 24217 tablet_service.cc:1467] Processing CreateTablet for tablet 0f2a4973d62c40df9ded5798c13452a6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d0f3ec9e77f34d0ca1399ecb757d7430]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 07:01:31.144096 24217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f2a4973d62c40df9ded5798c13452a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:01:31.150996 24427 tablet_bootstrap.cc:492] T 0f2a4973d62c40df9ded5798c13452a6 P 8b7aaeafa48249c7a5f29b048502bbe3: Bootstrap starting.
I20260807 07:01:31.152529 24434 tablet_bootstrap.cc:492] T 0f2a4973d62c40df9ded5798c13452a6 P a378a8686c4f4adc9bb13eda9783bb3d: Bootstrap starting.
I20260807 07:01:31.153604 24427 tablet_bootstrap.cc:654] T 0f2a4973d62c40df9ded5798c13452a6 P 8b7aaeafa48249c7a5f29b048502bbe3: Neither blocks nor log segments found. Creating new log.
I20260807 07:01:31.154247 24434 tablet_bootstrap.cc:654] T 0f2a4973d62c40df9ded5798c13452a6 P a378a8686c4f4adc9bb13eda9783bb3d: Neither blocks nor log segments found. Creating new log.
I20260807 07:01:31.157917 24431 tablet_bootstrap.cc:492] T 0f2a4973d62c40df9ded5798c13452a6 P 3d6807a83f544e32ae7874e242bf51ee: Bootstrap starting.
I20260807 07:01:31.160344 24431 tablet_bootstrap.cc:654] T 0f2a4973d62c40df9ded5798c13452a6 P 3d6807a83f544e32ae7874e242bf51ee: Neither blocks nor log segments found. Creating new log.
I20260807 07:01:31.162747 24427 tablet_bootstrap.cc:492] T 0f2a4973d62c40df9ded5798c13452a6 P 8b7aaeafa48249c7a5f29b048502bbe3: No bootstrap required, opened a new log
I20260807 07:01:31.163014 24427 ts_tablet_manager.cc:1397] T 0f2a4973d62c40df9ded5798c13452a6 P 8b7aaeafa48249c7a5f29b048502bbe3: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260807 07:01:31.163960 24427 raft_consensus.cc:348] T 0f2a4973d62c40df9ded5798c13452a6 P 8b7aaeafa48249c7a5f29b048502bbe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6807a83f544e32ae7874e242bf51ee" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42307 } } peers { permanent_uuid: "a378a8686c4f4adc9bb13eda9783bb3d" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40097 } } peers { permanent_uuid: "8b7aaeafa48249c7a5f29b048502bbe3" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40193 } }
I20260807 07:01:31.164337 24427 raft_consensus.cc:374] T 0f2a4973d62c40df9ded5798c13452a6 P 8b7aaeafa48249c7a5f29b048502bbe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:01:31.164484 24427 raft_consensus.cc:729] T 0f2a4973d62c40df9ded5798c13452a6 P 8b7aaeafa48249c7a5f29b048502bbe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b7aaeafa48249c7a5f29b048502bbe3, State: Initialized, Role: FOLLOWER
I20260807 07:01:31.164863 24427 consensus_queue.cc:260] T 0f2a4973d62c40df9ded5798c13452a6 P 8b7aaeafa48249c7a5f29b048502bbe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6807a83f544e32ae7874e242bf51ee" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42307 } } peers { permanent_uuid: "a378a8686c4f4adc9bb13eda9783bb3d" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40097 } } peers { permanent_uuid: "8b7aaeafa48249c7a5f29b048502bbe3" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40193 } }
I20260807 07:01:31.166121 24427 ts_tablet_manager.cc:1428] T 0f2a4973d62c40df9ded5798c13452a6 P 8b7aaeafa48249c7a5f29b048502bbe3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260807 07:01:31.166574 24431 tablet_bootstrap.cc:492] T 0f2a4973d62c40df9ded5798c13452a6 P 3d6807a83f544e32ae7874e242bf51ee: No bootstrap required, opened a new log
I20260807 07:01:31.166774 24431 ts_tablet_manager.cc:1397] T 0f2a4973d62c40df9ded5798c13452a6 P 3d6807a83f544e32ae7874e242bf51ee: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260807 07:01:31.167487 24431 raft_consensus.cc:348] T 0f2a4973d62c40df9ded5798c13452a6 P 3d6807a83f544e32ae7874e242bf51ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6807a83f544e32ae7874e242bf51ee" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42307 } } peers { permanent_uuid: "a378a8686c4f4adc9bb13eda9783bb3d" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40097 } } peers { permanent_uuid: "8b7aaeafa48249c7a5f29b048502bbe3" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40193 } }
I20260807 07:01:31.167729 24431 raft_consensus.cc:374] T 0f2a4973d62c40df9ded5798c13452a6 P 3d6807a83f544e32ae7874e242bf51ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:01:31.167802 24431 raft_consensus.cc:729] T 0f2a4973d62c40df9ded5798c13452a6 P 3d6807a83f544e32ae7874e242bf51ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d6807a83f544e32ae7874e242bf51ee, State: Initialized, Role: FOLLOWER
I20260807 07:01:31.167997 24431 consensus_queue.cc:260] T 0f2a4973d62c40df9ded5798c13452a6 P 3d6807a83f544e32ae7874e242bf51ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6807a83f544e32ae7874e242bf51ee" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42307 } } peers { permanent_uuid: "a378a8686c4f4adc9bb13eda9783bb3d" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40097 } } peers { permanent_uuid: "8b7aaeafa48249c7a5f29b048502bbe3" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40193 } }
I20260807 07:01:31.168526 24431 ts_tablet_manager.cc:1428] T 0f2a4973d62c40df9ded5798c13452a6 P 3d6807a83f544e32ae7874e242bf51ee: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260807 07:01:31.168396 24377 tablet_replica.cc:439] T 0f2a4973d62c40df9ded5798c13452a6 P 3d6807a83f544e32ae7874e242bf51ee: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6807a83f544e32ae7874e242bf51ee" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42307 } } peers { permanent_uuid: "a378a8686c4f4adc9bb13eda9783bb3d" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40097 } } peers { permanent_uuid: "8b7aaeafa48249c7a5f29b048502bbe3" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40193 } } }
I20260807 07:01:31.168758 24377 tablet_replica.cc:442] T 0f2a4973d62c40df9ded5798c13452a6 P 3d6807a83f544e32ae7874e242bf51ee: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 07:01:31.169144 24439 tablet_replica.cc:439] T 0f2a4973d62c40df9ded5798c13452a6 P 8b7aaeafa48249c7a5f29b048502bbe3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6807a83f544e32ae7874e242bf51ee" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42307 } } peers { permanent_uuid: "a378a8686c4f4adc9bb13eda9783bb3d" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40097 } } peers { permanent_uuid: "8b7aaeafa48249c7a5f29b048502bbe3" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40193 } } }
I20260807 07:01:31.169585 24439 tablet_replica.cc:442] T 0f2a4973d62c40df9ded5798c13452a6 P 8b7aaeafa48249c7a5f29b048502bbe3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 07:01:31.169792 24434 tablet_bootstrap.cc:492] T 0f2a4973d62c40df9ded5798c13452a6 P a378a8686c4f4adc9bb13eda9783bb3d: No bootstrap required, opened a new log
I20260807 07:01:31.170007 24434 ts_tablet_manager.cc:1397] T 0f2a4973d62c40df9ded5798c13452a6 P a378a8686c4f4adc9bb13eda9783bb3d: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.000s
I20260807 07:01:31.170751 24434 raft_consensus.cc:348] T 0f2a4973d62c40df9ded5798c13452a6 P a378a8686c4f4adc9bb13eda9783bb3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6807a83f544e32ae7874e242bf51ee" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42307 } } peers { permanent_uuid: "a378a8686c4f4adc9bb13eda9783bb3d" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40097 } } peers { permanent_uuid: "8b7aaeafa48249c7a5f29b048502bbe3" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40193 } }
I20260807 07:01:31.171013 24434 raft_consensus.cc:374] T 0f2a4973d62c40df9ded5798c13452a6 P a378a8686c4f4adc9bb13eda9783bb3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:01:31.171154 24434 raft_consensus.cc:729] T 0f2a4973d62c40df9ded5798c13452a6 P a378a8686c4f4adc9bb13eda9783bb3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a378a8686c4f4adc9bb13eda9783bb3d, State: Initialized, Role: FOLLOWER
I20260807 07:01:31.171401 24434 consensus_queue.cc:260] T 0f2a4973d62c40df9ded5798c13452a6 P a378a8686c4f4adc9bb13eda9783bb3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6807a83f544e32ae7874e242bf51ee" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42307 } } peers { permanent_uuid: "a378a8686c4f4adc9bb13eda9783bb3d" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40097 } } peers { permanent_uuid: "8b7aaeafa48249c7a5f29b048502bbe3" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40193 } }
I20260807 07:01:31.172219 24434 ts_tablet_manager.cc:1428] T 0f2a4973d62c40df9ded5798c13452a6 P a378a8686c4f4adc9bb13eda9783bb3d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260807 07:01:31.173508 24440 tablet_replica.cc:439] T 0f2a4973d62c40df9ded5798c13452a6 P a378a8686c4f4adc9bb13eda9783bb3d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6807a83f544e32ae7874e242bf51ee" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42307 } } peers { permanent_uuid: "a378a8686c4f4adc9bb13eda9783bb3d" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40097 } } peers { permanent_uuid: "8b7aaeafa48249c7a5f29b048502bbe3" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40193 } } }
I20260807 07:01:31.173839 24440 tablet_replica.cc:442] T 0f2a4973d62c40df9ded5798c13452a6 P a378a8686c4f4adc9bb13eda9783bb3d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 07:01:31.373668 24377 raft_consensus.cc:482] T 0f2a4973d62c40df9ded5798c13452a6 P 3d6807a83f544e32ae7874e242bf51ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:01:31.373932 24377 raft_consensus.cc:504] T 0f2a4973d62c40df9ded5798c13452a6 P 3d6807a83f544e32ae7874e242bf51ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6807a83f544e32ae7874e242bf51ee" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42307 } } peers { permanent_uuid: "a378a8686c4f4adc9bb13eda9783bb3d" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40097 } } peers { permanent_uuid: "8b7aaeafa48249c7a5f29b048502bbe3" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40193 } }
I20260807 07:01:31.374548 24377 leader_election.cc:288] T 0f2a4973d62c40df9ded5798c13452a6 P 3d6807a83f544e32ae7874e242bf51ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a378a8686c4f4adc9bb13eda9783bb3d (127.21.83.2:40097), 8b7aaeafa48249c7a5f29b048502bbe3 (127.21.83.3:40193)
I20260807 07:01:31.375288 24045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f2a4973d62c40df9ded5798c13452a6" candidate_uuid: "3d6807a83f544e32ae7874e242bf51ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a378a8686c4f4adc9bb13eda9783bb3d" is_pre_election: true
I20260807 07:01:31.375286 24251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f2a4973d62c40df9ded5798c13452a6" candidate_uuid: "3d6807a83f544e32ae7874e242bf51ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b7aaeafa48249c7a5f29b048502bbe3" is_pre_election: true
I20260807 07:01:31.375582 24045 raft_consensus.cc:2454] T 0f2a4973d62c40df9ded5798c13452a6 P a378a8686c4f4adc9bb13eda9783bb3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d6807a83f544e32ae7874e242bf51ee in term 0.
I20260807 07:01:31.375582 24251 raft_consensus.cc:2454] T 0f2a4973d62c40df9ded5798c13452a6 P 8b7aaeafa48249c7a5f29b048502bbe3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d6807a83f544e32ae7874e242bf51ee in term 0.
I20260807 07:01:31.376029 23711 leader_election.cc:302] T 0f2a4973d62c40df9ded5798c13452a6 P 3d6807a83f544e32ae7874e242bf51ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3d6807a83f544e32ae7874e242bf51ee, 8b7aaeafa48249c7a5f29b048502bbe3, a378a8686c4f4adc9bb13eda9783bb3d; no voters:
I20260807 07:01:31.376329 24377 raft_consensus.cc:2781] T 0f2a4973d62c40df9ded5798c13452a6 P 3d6807a83f544e32ae7874e242bf51ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 07:01:31.376423 24377 raft_consensus.cc:482] T 0f2a4973d62c40df9ded5798c13452a6 P 3d6807a83f544e32ae7874e242bf51ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:01:31.376545 24377 raft_consensus.cc:3037] T 0f2a4973d62c40df9ded5798c13452a6 P 3d6807a83f544e32ae7874e242bf51ee [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:01:31.378633 24377 raft_consensus.cc:504] T 0f2a4973d62c40df9ded5798c13452a6 P 3d6807a83f544e32ae7874e242bf51ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6807a83f544e32ae7874e242bf51ee" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42307 } } peers { permanent_uuid: "a378a8686c4f4adc9bb13eda9783bb3d" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40097 } } peers { permanent_uuid: "8b7aaeafa48249c7a5f29b048502bbe3" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40193 } }
I20260807 07:01:31.379243 24377 leader_election.cc:288] T 0f2a4973d62c40df9ded5798c13452a6 P 3d6807a83f544e32ae7874e242bf51ee [CANDIDATE]: Term 1 election: Requested vote from peers a378a8686c4f4adc9bb13eda9783bb3d (127.21.83.2:40097), 8b7aaeafa48249c7a5f29b048502bbe3 (127.21.83.3:40193)
I20260807 07:01:31.379710 24045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f2a4973d62c40df9ded5798c13452a6" candidate_uuid: "3d6807a83f544e32ae7874e242bf51ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a378a8686c4f4adc9bb13eda9783bb3d"
I20260807 07:01:31.379951 24045 raft_consensus.cc:3037] T 0f2a4973d62c40df9ded5798c13452a6 P a378a8686c4f4adc9bb13eda9783bb3d [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:01:31.379907 24251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f2a4973d62c40df9ded5798c13452a6" candidate_uuid: "3d6807a83f544e32ae7874e242bf51ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b7aaeafa48249c7a5f29b048502bbe3"
I20260807 07:01:31.380133 24251 raft_consensus.cc:3037] T 0f2a4973d62c40df9ded5798c13452a6 P 8b7aaeafa48249c7a5f29b048502bbe3 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:01:31.382473 24045 raft_consensus.cc:2454] T 0f2a4973d62c40df9ded5798c13452a6 P a378a8686c4f4adc9bb13eda9783bb3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d6807a83f544e32ae7874e242bf51ee in term 1.
I20260807 07:01:31.382699 24251 raft_consensus.cc:2454] T 0f2a4973d62c40df9ded5798c13452a6 P 8b7aaeafa48249c7a5f29b048502bbe3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d6807a83f544e32ae7874e242bf51ee in term 1.
I20260807 07:01:31.383056 23705 leader_election.cc:302] T 0f2a4973d62c40df9ded5798c13452a6 P 3d6807a83f544e32ae7874e242bf51ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d6807a83f544e32ae7874e242bf51ee, a378a8686c4f4adc9bb13eda9783bb3d; no voters:
I20260807 07:01:31.383405 24377 raft_consensus.cc:2781] T 0f2a4973d62c40df9ded5798c13452a6 P 3d6807a83f544e32ae7874e242bf51ee [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:01:31.383647 24377 raft_consensus.cc:686] T 0f2a4973d62c40df9ded5798c13452a6 P 3d6807a83f544e32ae7874e242bf51ee [term 1 LEADER]: Becoming Leader. State: Replica: 3d6807a83f544e32ae7874e242bf51ee, State: Running, Role: LEADER
I20260807 07:01:31.384076 24377 consensus_queue.cc:237] T 0f2a4973d62c40df9ded5798c13452a6 P 3d6807a83f544e32ae7874e242bf51ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6807a83f544e32ae7874e242bf51ee" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42307 } } peers { permanent_uuid: "a378a8686c4f4adc9bb13eda9783bb3d" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40097 } } peers { permanent_uuid: "8b7aaeafa48249c7a5f29b048502bbe3" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40193 } }
I20260807 07:01:31.386194 24407 tablet_replica.cc:439] T 0f2a4973d62c40df9ded5798c13452a6 P 3d6807a83f544e32ae7874e242bf51ee: TxnStatusTablet state changed. Reason: New leader 3d6807a83f544e32ae7874e242bf51ee. Latest consensus state: current_term: 1 leader_uuid: "3d6807a83f544e32ae7874e242bf51ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6807a83f544e32ae7874e242bf51ee" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42307 } } peers { permanent_uuid: "a378a8686c4f4adc9bb13eda9783bb3d" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40097 } } peers { permanent_uuid: "8b7aaeafa48249c7a5f29b048502bbe3" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40193 } } }
I20260807 07:01:31.386708 24407 tablet_replica.cc:442] T 0f2a4973d62c40df9ded5798c13452a6 P 3d6807a83f544e32ae7874e242bf51ee: This TxnStatusTablet replica's current role is: LEADER
I20260807 07:01:31.387521 24443 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 07:01:31.387848 24443 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260807 07:01:31.387770 23564 catalog_manager.cc:5495] T 0f2a4973d62c40df9ded5798c13452a6 P 3d6807a83f544e32ae7874e242bf51ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d6807a83f544e32ae7874e242bf51ee (127.21.83.1). New cstate: current_term: 1 leader_uuid: "3d6807a83f544e32ae7874e242bf51ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6807a83f544e32ae7874e242bf51ee" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a378a8686c4f4adc9bb13eda9783bb3d" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b7aaeafa48249c7a5f29b048502bbe3" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40193 } health_report { overall_health: UNKNOWN } } }
I20260807 07:01:31.420680 24407 consensus_queue.cc:1035] T 7dee679534a045da9f0c97e014a15065 P 3d6807a83f544e32ae7874e242bf51ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b7aaeafa48249c7a5f29b048502bbe3" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:01:31.426541 24407 consensus_queue.cc:1035] T 7dee679534a045da9f0c97e014a15065 P 3d6807a83f544e32ae7874e242bf51ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "a378a8686c4f4adc9bb13eda9783bb3d" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:01:31.455510 24425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:31.450145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:42307 (local address 127.0.0.1:35212)
0807 07:01:31.450372 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:31.450393 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:31.450532 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:31.451144 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:31.451149 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:31.451162 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 07:01:31.451355 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:31.451366 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:31.452489 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:31.452495 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:31.453571 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:31.453584 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:31.454307 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:31.454314 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:31.454483 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:31.454543 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 07:01:31.454935 (+ 392us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 07:01:31.454958 (+ 23us) client_negotiation.cc:769] Sending connection context
0807 07:01:31.455029 (+ 71us) client_negotiation.cc:240] Negotiation successful
0807 07:01:31.455192 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260807 07:01:31.455565 24452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:31.450261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35212 (local address 127.21.83.1:42307)
0807 07:01:31.450854 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:31.450858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:31.450875 (+ 17us) server_negotiation.cc:407] Connection header received
0807 07:01:31.450935 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:31.450938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:31.451004 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:31.451093 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:01:31.451519 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:31.452358 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:31.453708 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:31.454169 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:31.454288 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:31.454731 (+ 443us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:01:31.454837 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:01:31.454918 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:31.455062 (+ 144us) server_negotiation.cc:299] Negotiation successful
0807 07:01:31.455146 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":308,"threads_started":1}
I20260807 07:01:31.459215 24251 raft_consensus.cc:1261] T 0f2a4973d62c40df9ded5798c13452a6 P 8b7aaeafa48249c7a5f29b048502bbe3 [term 1 FOLLOWER]: Refusing update from remote peer 3d6807a83f544e32ae7874e242bf51ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:01:31.459285 24045 raft_consensus.cc:1261] T 0f2a4973d62c40df9ded5798c13452a6 P a378a8686c4f4adc9bb13eda9783bb3d [term 1 FOLLOWER]: Refusing update from remote peer 3d6807a83f544e32ae7874e242bf51ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:01:31.459957 24377 consensus_queue.cc:1035] T 0f2a4973d62c40df9ded5798c13452a6 P 3d6807a83f544e32ae7874e242bf51ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b7aaeafa48249c7a5f29b048502bbe3" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:01:31.460273 24407 consensus_queue.cc:1035] T 0f2a4973d62c40df9ded5798c13452a6 P 3d6807a83f544e32ae7874e242bf51ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "a378a8686c4f4adc9bb13eda9783bb3d" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:01:31.463322 24440 tablet_replica.cc:439] T 0f2a4973d62c40df9ded5798c13452a6 P a378a8686c4f4adc9bb13eda9783bb3d: TxnStatusTablet state changed. Reason: New leader 3d6807a83f544e32ae7874e242bf51ee. Latest consensus state: current_term: 1 leader_uuid: "3d6807a83f544e32ae7874e242bf51ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6807a83f544e32ae7874e242bf51ee" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42307 } } peers { permanent_uuid: "a378a8686c4f4adc9bb13eda9783bb3d" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40097 } } peers { permanent_uuid: "8b7aaeafa48249c7a5f29b048502bbe3" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40193 } } }
I20260807 07:01:31.463601 24440 tablet_replica.cc:442] T 0f2a4973d62c40df9ded5798c13452a6 P a378a8686c4f4adc9bb13eda9783bb3d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 07:01:31.465621 24446 mvcc.cc:205] Tried to move back new op lower bound from 7315808630610788352 to 7315808630312071168. Current Snapshot: MvccSnapshot[applied={T|T < 7315808630610788352}]
I20260807 07:01:31.467128 24440 tablet_replica.cc:439] T 0f2a4973d62c40df9ded5798c13452a6 P a378a8686c4f4adc9bb13eda9783bb3d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3d6807a83f544e32ae7874e242bf51ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6807a83f544e32ae7874e242bf51ee" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42307 } } peers { permanent_uuid: "a378a8686c4f4adc9bb13eda9783bb3d" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40097 } } peers { permanent_uuid: "8b7aaeafa48249c7a5f29b048502bbe3" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40193 } } }
I20260807 07:01:31.467372 24440 tablet_replica.cc:442] T 0f2a4973d62c40df9ded5798c13452a6 P a378a8686c4f4adc9bb13eda9783bb3d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 07:01:31.469599 24407 tablet_replica.cc:439] T 0f2a4973d62c40df9ded5798c13452a6 P 3d6807a83f544e32ae7874e242bf51ee: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3d6807a83f544e32ae7874e242bf51ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6807a83f544e32ae7874e242bf51ee" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42307 } } peers { permanent_uuid: "a378a8686c4f4adc9bb13eda9783bb3d" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40097 } } peers { permanent_uuid: "8b7aaeafa48249c7a5f29b048502bbe3" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40193 } } }
I20260807 07:01:31.470003 24407 tablet_replica.cc:442] T 0f2a4973d62c40df9ded5798c13452a6 P 3d6807a83f544e32ae7874e242bf51ee: This TxnStatusTablet replica's current role is: LEADER
I20260807 07:01:31.474956 24377 tablet_replica.cc:439] T 0f2a4973d62c40df9ded5798c13452a6 P 3d6807a83f544e32ae7874e242bf51ee: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3d6807a83f544e32ae7874e242bf51ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6807a83f544e32ae7874e242bf51ee" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42307 } } peers { permanent_uuid: "a378a8686c4f4adc9bb13eda9783bb3d" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40097 } } peers { permanent_uuid: "8b7aaeafa48249c7a5f29b048502bbe3" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40193 } } }
I20260807 07:01:31.475340 24377 tablet_replica.cc:442] T 0f2a4973d62c40df9ded5798c13452a6 P 3d6807a83f544e32ae7874e242bf51ee: This TxnStatusTablet replica's current role is: LEADER
I20260807 07:01:31.476162 24439 tablet_replica.cc:439] T 0f2a4973d62c40df9ded5798c13452a6 P 8b7aaeafa48249c7a5f29b048502bbe3: TxnStatusTablet state changed. Reason: New leader 3d6807a83f544e32ae7874e242bf51ee. Latest consensus state: current_term: 1 leader_uuid: "3d6807a83f544e32ae7874e242bf51ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6807a83f544e32ae7874e242bf51ee" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42307 } } peers { permanent_uuid: "a378a8686c4f4adc9bb13eda9783bb3d" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40097 } } peers { permanent_uuid: "8b7aaeafa48249c7a5f29b048502bbe3" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40193 } } }
I20260807 07:01:31.476469 24439 tablet_replica.cc:442] T 0f2a4973d62c40df9ded5798c13452a6 P 8b7aaeafa48249c7a5f29b048502bbe3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 07:01:31.477912 24439 tablet_replica.cc:439] T 0f2a4973d62c40df9ded5798c13452a6 P 8b7aaeafa48249c7a5f29b048502bbe3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3d6807a83f544e32ae7874e242bf51ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6807a83f544e32ae7874e242bf51ee" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42307 } } peers { permanent_uuid: "a378a8686c4f4adc9bb13eda9783bb3d" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40097 } } peers { permanent_uuid: "8b7aaeafa48249c7a5f29b048502bbe3" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40193 } } }
I20260807 07:01:31.478178 24439 tablet_replica.cc:442] T 0f2a4973d62c40df9ded5798c13452a6 P 8b7aaeafa48249c7a5f29b048502bbe3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 07:01:31.481317 24450 mvcc.cc:205] Tried to move back new op lower bound from 7315808630610788352 to 7315808630312071168. Current Snapshot: MvccSnapshot[applied={T|T < 7315808630610788352}]
I20260807 07:01:31.499279 24452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:31.491855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:44675 (local address 127.21.83.1:42307)
0807 07:01:31.492030 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:31.492038 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:31.492608 (+ 570us) server_negotiation.cc:407] Connection header received
0807 07:01:31.492811 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:31.492816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:31.492930 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:31.493031 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:01:31.493709 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:31.494866 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:31.497063 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:31.497751 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:31.497888 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:31.498464 (+ 576us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:01:31.498604 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:01:31.498697 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:31.498831 (+ 134us) server_negotiation.cc:299] Negotiation successful
0807 07:01:31.498919 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":42}
I20260807 07:01:31.499614 24465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:31.491851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:42307 (local address 127.21.83.1:44675)
0807 07:01:31.492491 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:31.492511 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:31.492682 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:31.493060 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:31.493065 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:31.493079 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 07:01:31.493542 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:31.493556 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:31.495028 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:31.495033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:31.496872 (+ 1839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:31.496928 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:31.497955 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:31.497962 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:31.498172 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:31.498269 (+ 97us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 07:01:31.498714 (+ 445us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 07:01:31.498726 (+ 12us) client_negotiation.cc:769] Sending connection context
0807 07:01:31.498810 (+ 84us) client_negotiation.cc:240] Negotiation successful
0807 07:01:31.498972 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"mutex_wait_us":22,"thread_start_us":296,"threads_started":1}
I20260807 07:01:31.573247 23565 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:01:31.573491 23565 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:01:31.576742 23565 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e5bf836233d747769243d412d635e83f: Sending DeleteTablet for 3 replicas of tablet 7dee679534a045da9f0c97e014a15065
I20260807 07:01:31.577732 24011 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dee679534a045da9f0c97e014a15065 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:01:31 UTC) from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:56856
I20260807 07:01:31.577732 23786 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dee679534a045da9f0c97e014a15065 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:01:31 UTC) from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:35192
I20260807 07:01:31.577981 24217 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dee679534a045da9f0c97e014a15065 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:01:31 UTC) from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:39612
I20260807 07:01:31.589025 24471 tablet_replica.cc:330] T 7dee679534a045da9f0c97e014a15065 P a378a8686c4f4adc9bb13eda9783bb3d: stopping tablet replica
I20260807 07:01:31.589469 24471 raft_consensus.cc:2229] T 7dee679534a045da9f0c97e014a15065 P a378a8686c4f4adc9bb13eda9783bb3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:01:31.589900 24471 raft_consensus.cc:2258] T 7dee679534a045da9f0c97e014a15065 P a378a8686c4f4adc9bb13eda9783bb3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:01:31.591476 24471 ts_tablet_manager.cc:1905] T 7dee679534a045da9f0c97e014a15065 P a378a8686c4f4adc9bb13eda9783bb3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:01:31.591622 24475 tablet_replica.cc:330] T 7dee679534a045da9f0c97e014a15065 P 8b7aaeafa48249c7a5f29b048502bbe3: stopping tablet replica
I20260807 07:01:31.592387 24475 raft_consensus.cc:2229] T 7dee679534a045da9f0c97e014a15065 P 8b7aaeafa48249c7a5f29b048502bbe3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:01:31.593071 24475 raft_consensus.cc:2258] T 7dee679534a045da9f0c97e014a15065 P 8b7aaeafa48249c7a5f29b048502bbe3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:01:31.594727 24475 ts_tablet_manager.cc:1905] T 7dee679534a045da9f0c97e014a15065 P 8b7aaeafa48249c7a5f29b048502bbe3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:01:31.598708 24473 tablet_replica.cc:330] T 7dee679534a045da9f0c97e014a15065 P 3d6807a83f544e32ae7874e242bf51ee: stopping tablet replica
I20260807 07:01:31.598958 24473 raft_consensus.cc:2229] T 7dee679534a045da9f0c97e014a15065 P 3d6807a83f544e32ae7874e242bf51ee [term 1 LEADER]: Raft consensus shutting down.
I20260807 07:01:31.599459 24473 raft_consensus.cc:2258] T 7dee679534a045da9f0c97e014a15065 P 3d6807a83f544e32ae7874e242bf51ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:01:31.600629 24473 ts_tablet_manager.cc:1905] T 7dee679534a045da9f0c97e014a15065 P 3d6807a83f544e32ae7874e242bf51ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:01:31.603468 24475 ts_tablet_manager.cc:1918] T 7dee679534a045da9f0c97e014a15065 P 8b7aaeafa48249c7a5f29b048502bbe3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 07:01:31.603653 24475 log.cc:1196] T 7dee679534a045da9f0c97e014a15065 P 8b7aaeafa48249c7a5f29b048502bbe3: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-2/wal/wals/7dee679534a045da9f0c97e014a15065
I20260807 07:01:31.604131 24475 ts_tablet_manager.cc:1939] T 7dee679534a045da9f0c97e014a15065 P 8b7aaeafa48249c7a5f29b048502bbe3: Deleting consensus metadata
I20260807 07:01:31.605370 23546 catalog_manager.cc:4841] TS 8b7aaeafa48249c7a5f29b048502bbe3 (127.21.83.3:40193): tablet 7dee679534a045da9f0c97e014a15065 (table test-table [id=70f71a645e9e444bbb50c1cee0ebf978]) successfully deleted
I20260807 07:01:31.607242 24471 ts_tablet_manager.cc:1918] T 7dee679534a045da9f0c97e014a15065 P a378a8686c4f4adc9bb13eda9783bb3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 07:01:31.607360 24471 log.cc:1196] T 7dee679534a045da9f0c97e014a15065 P a378a8686c4f4adc9bb13eda9783bb3d: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-1/wal/wals/7dee679534a045da9f0c97e014a15065
I20260807 07:01:31.607772 24471 ts_tablet_manager.cc:1939] T 7dee679534a045da9f0c97e014a15065 P a378a8686c4f4adc9bb13eda9783bb3d: Deleting consensus metadata
I20260807 07:01:31.608701 23543 catalog_manager.cc:4841] TS a378a8686c4f4adc9bb13eda9783bb3d (127.21.83.2:40097): tablet 7dee679534a045da9f0c97e014a15065 (table test-table [id=70f71a645e9e444bbb50c1cee0ebf978]) successfully deleted
I20260807 07:01:31.611471 24473 ts_tablet_manager.cc:1918] T 7dee679534a045da9f0c97e014a15065 P 3d6807a83f544e32ae7874e242bf51ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 07:01:31.611617 24473 log.cc:1196] T 7dee679534a045da9f0c97e014a15065 P 3d6807a83f544e32ae7874e242bf51ee: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786086078080656-21836-0/minicluster-data/ts-0/wal/wals/7dee679534a045da9f0c97e014a15065
I20260807 07:01:31.612138 24473 ts_tablet_manager.cc:1939] T 7dee679534a045da9f0c97e014a15065 P 3d6807a83f544e32ae7874e242bf51ee: Deleting consensus metadata
Aug 07 07:01:31 dist-test-slave-ncp9 krb5kdc[23502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086090, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.83.1@KRBTEST.COM
I20260807 07:01:31.613545 23545 catalog_manager.cc:4841] TS 3d6807a83f544e32ae7874e242bf51ee (127.21.83.1:42307): tablet 7dee679534a045da9f0c97e014a15065 (table test-table [id=70f71a645e9e444bbb50c1cee0ebf978]) successfully deleted
I20260807 07:01:31.620221 24452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:31.591885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35226 (local address 127.21.83.1:42307)
0807 07:01:31.592136 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:31.592143 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:31.601402 (+ 9259us) server_negotiation.cc:407] Connection header received
0807 07:01:31.601628 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:31.601634 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:31.601723 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:31.601798 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:31.604488 (+ 2690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:31.605705 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:31.607024 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:31.609264 (+ 2240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:31.609404 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:31.614415 (+ 5011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:31.614429 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:31.614445 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:31.614507 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:31.617690 (+ 3183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:31.618399 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:31.618405 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:31.618407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:31.618548 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:31.618907 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:31.618910 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:31.618912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:31.619238 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:31.619479 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:31.619780 (+ 301us) server_negotiation.cc:299] Negotiation successful
0807 07:01:31.619931 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260807 07:01:31.621050 23894 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35226
I20260807 07:01:31.636991 24424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:31.623909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51164 (local address 127.21.83.62:33131)
0807 07:01:31.624094 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:31.624100 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:31.624294 (+ 194us) server_negotiation.cc:407] Connection header received
0807 07:01:31.624451 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:31.624455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:31.624516 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:31.624592 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:31.626129 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:31.626842 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:31.628550 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:31.629086 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:31.629183 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:31.631936 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:31.631940 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:31.631943 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:31.631975 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:31.634574 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:31.635330 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:31.635333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:31.635336 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:31.635447 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:31.635776 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:31.635779 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:31.635781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:31.636044 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:31.636218 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:31.636531 (+ 313us) server_negotiation.cc:299] Negotiation successful
0807 07:01:31.636709 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260807 07:01:31.637599 23565 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51164
I20260807 07:01:31.639423 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 23672
I20260807 07:01:31.653050 24452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:31.640237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35234 (local address 127.21.83.1:42307)
0807 07:01:31.640798 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:31.640804 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:31.640822 (+ 18us) server_negotiation.cc:407] Connection header received
0807 07:01:31.641043 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:31.641048 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:31.641117 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:31.641201 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:31.642301 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:31.643061 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:31.644361 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:31.644964 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:31.645080 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:31.647193 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:31.647198 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:31.647202 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:31.647239 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:31.650526 (+ 3287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:31.651087 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:31.651093 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:31.651097 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:31.651253 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:31.651589 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:31.651594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:31.651598 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:31.651893 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:31.652212 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:31.652463 (+ 251us) server_negotiation.cc:299] Negotiation successful
0807 07:01:31.652709 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421}
W20260807 07:01:31.653585 23894 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35234
W20260807 07:01:31.654093 21836 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 23672: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 07:01:31.654255 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 23672
I20260807 07:01:31.685897 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 23909
Aug 07 07:01:31 dist-test-slave-ncp9 krb5kdc[23502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086090, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.83.2@KRBTEST.COM
I20260807 07:01:31.699434 24482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:31.686620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56870 (local address 127.21.83.2:40097)
0807 07:01:31.687101 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:31.687106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:31.687123 (+ 17us) server_negotiation.cc:407] Connection header received
0807 07:01:31.687224 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:31.687228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:31.687286 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:31.687371 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:31.688447 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:31.689204 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:31.690387 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:31.690818 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:31.690906 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:31.694679 (+ 3773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:31.694699 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:31.694718 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:31.694761 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:31.697310 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:31.697854 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:31.697860 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:31.697862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:31.697981 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:31.698241 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:31.698244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:31.698246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:31.698561 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:31.698788 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:31.698938 (+ 150us) server_negotiation.cc:299] Negotiation successful
0807 07:01:31.699110 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":234,"threads_started":1}
W20260807 07:01:31.700198 24106 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56870
W20260807 07:01:31.700742 21836 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 23909: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 07:01:31.700881 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 23909
I20260807 07:01:31.723234 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 24116
Aug 07 07:01:31 dist-test-slave-ncp9 krb5kdc[23502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086090, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.83.3@KRBTEST.COM
I20260807 07:01:31.737351 24483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:31.723939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39614 (local address 127.21.83.3:40193)
0807 07:01:31.724389 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:31.724393 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:31.724410 (+ 17us) server_negotiation.cc:407] Connection header received
0807 07:01:31.724465 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:31.724468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:31.724546 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:31.724635 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:31.725929 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:31.726673 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:31.727947 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:31.728369 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:31.728464 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:31.732408 (+ 3944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:31.732422 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:31.732437 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:31.732470 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:31.735128 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:31.735695 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:31.735701 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:31.735704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:31.735833 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:31.736105 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:31.736108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:31.736109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:31.736414 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:31.736652 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:31.736810 (+ 158us) server_negotiation.cc:299] Negotiation successful
0807 07:01:31.737072 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":249,"threads_started":1}
W20260807 07:01:31.737970 24322 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39614
W20260807 07:01:31.738561 21836 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 24116: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 07:01:31.738689 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 24116
I20260807 07:01:31.761694 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 23521
I20260807 07:01:31.762737 23621 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:01:31.879765 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 23521
2026-08-07T07:01:31Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5212 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 07:01:31 dist-test-slave-ncp9 krb5kdc[24491](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 07:01:31 dist-test-slave-ncp9 krb5kdc[24491](info): set up 2 sockets
Aug 07 07:01:31 dist-test-slave-ncp9 krb5kdc[24491](info): commencing operation
krb5kdc: starting...
W20260807 07:01:33.051574 21836 mini_kdc.cc:121] Time spent starting KDC: real 1.124s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Aug 07 07:01:33 dist-test-slave-ncp9 krb5kdc[24491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T07:01:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:01:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
I20260807 07:01:33.280085 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:34197
--webserver_interface=127.21.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:42185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:34197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:01:33.623556 24511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:01:33.623919 24511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:01:33.624063 24511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:01:33.631667 24511 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:01:33.631803 24511 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260807 07:01:33.631876 24511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:01:33.631966 24511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:01:33.632097 24511 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:01:33.632275 24511 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:01:33.642876 24511 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:42185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:34197
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:34197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:01:33.644246 24511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:01:33.645664 24511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:01:33.657930 24527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:33.657979 24524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:33.657907 24525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:01:33.658674 24511 server_base.cc:1034] running on GCE node
I20260807 07:01:33.659504 24511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:01:33.661104 24511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:01:33.662334 24511 hybrid_clock.cc:648] HybridClock initialized: now 1786086093662251 us; error 57 us; skew 500 ppm
I20260807 07:01:33.662686 24511 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:01:33 dist-test-slave-ncp9 krb5kdc[24491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086093, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:01:33.666074 24511 init.cc:377] Logged in from keytab as kudu/127.21.83.62@KRBTEST.COM (short username kudu)
I20260807 07:01:33.667734 24511 webserver.cc:460] Webserver started at http://127.21.83.62:43221/ using document root <none> and password file <none>
I20260807 07:01:33.668481 24511 fs_manager.cc:359] Metadata directory not provided
I20260807 07:01:33.668599 24511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:01:33.668967 24511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:01:33.672130 24511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "bd7c08a8e2434a539ae91b39b7c750dc"
format_stamp: "Formatted at 2026-08-07 07:01:33 on dist-test-slave-ncp9"
server_key: "12b70b5c4dba1859c24566e8c48d05c7"
server_key_iv: "161c975a7e19f4ec9b3c2a43e07d7492"
server_key_version: "encryptionkey@0"
I20260807 07:01:33.672991 24511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "bd7c08a8e2434a539ae91b39b7c750dc"
format_stamp: "Formatted at 2026-08-07 07:01:33 on dist-test-slave-ncp9"
server_key: "12b70b5c4dba1859c24566e8c48d05c7"
server_key_iv: "161c975a7e19f4ec9b3c2a43e07d7492"
server_key_version: "encryptionkey@0"
I20260807 07:01:33.678342 24511 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 07:01:33.681917 24536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:33.683288 24511 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 07:01:33.683468 24511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "bd7c08a8e2434a539ae91b39b7c750dc"
format_stamp: "Formatted at 2026-08-07 07:01:33 on dist-test-slave-ncp9"
server_key: "12b70b5c4dba1859c24566e8c48d05c7"
server_key_iv: "161c975a7e19f4ec9b3c2a43e07d7492"
server_key_version: "encryptionkey@0"
I20260807 07:01:33.683704 24511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:01:33.704389 24511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:01:33.708715 24511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:01:33.709018 24511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:01:33.731144 24511 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.62:34197
I20260807 07:01:33.731166 24628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.62:34197 every 8 connection(s)
I20260807 07:01:33.732568 24511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:01:33.733227 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 24511
I20260807 07:01:33.733417 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:01:33.734226 21836 external_mini_cluster.cc:1428] Setting key 389d217667903273e86f4cc2eea72fed
I20260807 07:01:33.737123 24629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:01:33.748217 24629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd7c08a8e2434a539ae91b39b7c750dc: Bootstrap starting.
Aug 07 07:01:33 dist-test-slave-ncp9 krb5kdc[24491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:01:33.751639 24629 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd7c08a8e2434a539ae91b39b7c750dc: Neither blocks nor log segments found. Creating new log.
I20260807 07:01:33.752825 24629 log.cc:824] T 00000000000000000000000000000000 P bd7c08a8e2434a539ae91b39b7c750dc: Log is configured to *not* fsync() on all Append() calls
I20260807 07:01:33.756466 24629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd7c08a8e2434a539ae91b39b7c750dc: No bootstrap required, opened a new log
I20260807 07:01:33.758044 24632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:33.736549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37274 (local address 127.21.83.62:34197)
0807 07:01:33.737457 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:33.737469 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:33.737512 (+ 43us) server_negotiation.cc:407] Connection header received
0807 07:01:33.739198 (+ 1686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:33.739221 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:33.739652 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:33.740042 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:33.741909 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:33.743419 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:33.744747 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:33.745538 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:33.745733 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:33.749871 (+ 4138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:33.749880 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:33.749896 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:33.749946 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:33.753086 (+ 3140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:33.754054 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:33.754062 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:33.754072 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:33.754280 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:33.754679 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:33.754684 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:33.754687 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:33.755447 (+ 760us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:33.755751 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:33.756170 (+ 419us) server_negotiation.cc:299] Negotiation successful
0807 07:01:33.756626 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":420,"threads_started":1}
I20260807 07:01:33.765520 24629 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd7c08a8e2434a539ae91b39b7c750dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7c08a8e2434a539ae91b39b7c750dc" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 34197 } }
I20260807 07:01:33.766017 24629 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd7c08a8e2434a539ae91b39b7c750dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:01:33.766139 24629 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd7c08a8e2434a539ae91b39b7c750dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd7c08a8e2434a539ae91b39b7c750dc, State: Initialized, Role: FOLLOWER
I20260807 07:01:33.766798 24629 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd7c08a8e2434a539ae91b39b7c750dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7c08a8e2434a539ae91b39b7c750dc" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 34197 } }
I20260807 07:01:33.767032 24629 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd7c08a8e2434a539ae91b39b7c750dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:01:33.767170 24629 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd7c08a8e2434a539ae91b39b7c750dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:01:33.767341 24629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd7c08a8e2434a539ae91b39b7c750dc [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:01:33.769919 24629 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd7c08a8e2434a539ae91b39b7c750dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7c08a8e2434a539ae91b39b7c750dc" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 34197 } }
I20260807 07:01:33.770473 24629 leader_election.cc:302] T 00000000000000000000000000000000 P bd7c08a8e2434a539ae91b39b7c750dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd7c08a8e2434a539ae91b39b7c750dc; no voters:
I20260807 07:01:33.771078 24629 leader_election.cc:288] T 00000000000000000000000000000000 P bd7c08a8e2434a539ae91b39b7c750dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 07:01:33.771226 24636 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd7c08a8e2434a539ae91b39b7c750dc [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:01:33.771926 24636 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd7c08a8e2434a539ae91b39b7c750dc [term 1 LEADER]: Becoming Leader. State: Replica: bd7c08a8e2434a539ae91b39b7c750dc, State: Running, Role: LEADER
I20260807 07:01:33.772505 24636 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd7c08a8e2434a539ae91b39b7c750dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7c08a8e2434a539ae91b39b7c750dc" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 34197 } }
I20260807 07:01:33.773162 24629 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd7c08a8e2434a539ae91b39b7c750dc [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:01:33.775410 24637 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd7c08a8e2434a539ae91b39b7c750dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd7c08a8e2434a539ae91b39b7c750dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7c08a8e2434a539ae91b39b7c750dc" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 34197 } } }
I20260807 07:01:33.775926 24637 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd7c08a8e2434a539ae91b39b7c750dc [sys.catalog]: This master's current role is: LEADER
I20260807 07:01:33.775518 24640 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd7c08a8e2434a539ae91b39b7c750dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd7c08a8e2434a539ae91b39b7c750dc. Latest consensus state: current_term: 1 leader_uuid: "bd7c08a8e2434a539ae91b39b7c750dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7c08a8e2434a539ae91b39b7c750dc" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 34197 } } }
I20260807 07:01:33.776881 24640 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd7c08a8e2434a539ae91b39b7c750dc [sys.catalog]: This master's current role is: LEADER
I20260807 07:01:33.778304 24647 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:01:33.784060 24647 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:01:33.794097 24647 catalog_manager.cc:1347] Generated new cluster ID: 2c95373bbab84c83be4ffa98bd2689ae
I20260807 07:01:33.794220 24647 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:01:33.819392 24647 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:01:33.820814 24647 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:01:33.827209 24647 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd7c08a8e2434a539ae91b39b7c750dc: Generated new TSK 0
I20260807 07:01:33.828105 24647 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.83.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab.
Entry for principal kudu/127.21.83.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab.
I20260807 07:01:33.890640 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.1:0
--local_ip_for_outbound_sockets=127.21.83.1
--webserver_interface=127.21.83.1
--webserver_port=0
--tserver_master_addrs=127.21.83.62:34197
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab
--principal=kudu/127.21.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:42185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:01:34.254007 24664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:01:34.254330 24664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:01:34.254478 24664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:01:34.262121 24664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:01:34.262276 24664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:01:34.262409 24664 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 07:01:34.262472 24664 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 07:01:34.262579 24664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.1
I20260807 07:01:34.272799 24664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:42185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab
--principal=kudu/127.21.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.83.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.83.62:34197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.1
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:01:34.274276 24664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:01:34.275852 24664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:01:34.289322 24669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:34.289350 24670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:34.289285 24673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:01:34.290588 24664 server_base.cc:1034] running on GCE node
I20260807 07:01:34.291232 24664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:01:34.292207 24664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:01:34.293412 24664 hybrid_clock.cc:648] HybridClock initialized: now 1786086094293352 us; error 52 us; skew 500 ppm
I20260807 07:01:34.293721 24664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:01:34 dist-test-slave-ncp9 krb5kdc[24491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086094, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:01:34.296974 24664 init.cc:377] Logged in from keytab as kudu/127.21.83.1@KRBTEST.COM (short username kudu)
I20260807 07:01:34.298547 24664 webserver.cc:460] Webserver started at http://127.21.83.1:39809/ using document root <none> and password file <none>
I20260807 07:01:34.299223 24664 fs_manager.cc:359] Metadata directory not provided
I20260807 07:01:34.299330 24664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:01:34.299603 24664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:01:34.302736 24664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-0/data/instance:
uuid: "62aacfc55612476eac88f5a78966f57e"
format_stamp: "Formatted at 2026-08-07 07:01:34 on dist-test-slave-ncp9"
server_key: "7986102f5d3cd19adaf6e6a03ddd8092"
server_key_iv: "80d2c61f7d3496cf539954028c30b16f"
server_key_version: "encryptionkey@0"
I20260807 07:01:34.303555 24664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-0/wal/instance:
uuid: "62aacfc55612476eac88f5a78966f57e"
format_stamp: "Formatted at 2026-08-07 07:01:34 on dist-test-slave-ncp9"
server_key: "7986102f5d3cd19adaf6e6a03ddd8092"
server_key_iv: "80d2c61f7d3496cf539954028c30b16f"
server_key_version: "encryptionkey@0"
I20260807 07:01:34.309018 24664 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 07:01:34.312956 24683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:34.314574 24664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 07:01:34.314777 24664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-0/wal
uuid: "62aacfc55612476eac88f5a78966f57e"
format_stamp: "Formatted at 2026-08-07 07:01:34 on dist-test-slave-ncp9"
server_key: "7986102f5d3cd19adaf6e6a03ddd8092"
server_key_iv: "80d2c61f7d3496cf539954028c30b16f"
server_key_version: "encryptionkey@0"
I20260807 07:01:34.314993 24664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:01:34.334693 24664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:01:34.339388 24664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:01:34.339689 24664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:01:34.342686 24664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:01:34.342823 24664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:34.342965 24664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:01:34.343050 24664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 07:01:34 dist-test-slave-ncp9 krb5kdc[24491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086094, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.1@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:01:34.366511 24700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:34.343163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:46389 (local address 127.21.83.62:34197)
0807 07:01:34.343633 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:34.343638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:34.344976 (+ 1338us) server_negotiation.cc:407] Connection header received
0807 07:01:34.347151 (+ 2175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:34.347157 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:34.347255 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:34.347356 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:34.349911 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:34.350942 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:34.353497 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:34.354061 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:34.354213 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:34.359120 (+ 4907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:34.359127 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:34.359131 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:34.359175 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:34.362376 (+ 3201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:34.363569 (+ 1193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:34.363575 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:34.363579 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:34.363776 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:34.364269 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:34.364274 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:34.364278 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:34.364728 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:34.364922 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:34.365789 (+ 867us) server_negotiation.cc:299] Negotiation successful
0807 07:01:34.366077 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":240,"threads_started":1}
I20260807 07:01:34.367374 24702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:34.343907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:34197 (local address 127.21.83.1:46389)
0807 07:01:34.344778 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:34.344828 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:34.346824 (+ 1996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:34.347688 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:34.347702 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:34.348630 (+ 928us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:01:34.349617 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:34.349651 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:34.351134 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:34.351140 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:34.353265 (+ 2125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:34.353289 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:34.354330 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:34.354338 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:34.354589 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:34.355386 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:01:34.355411 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:01:34.358743 (+ 3332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:01:34.362615 (+ 3872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:34.362628 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:34.362653 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:34.363369 (+ 716us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:34.363907 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:34.363911 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:34.363913 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:34.364147 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:34.364940 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:01:34.364954 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:01:34.365363 (+ 409us) client_negotiation.cc:769] Sending connection context
0807 07:01:34.365623 (+ 260us) client_negotiation.cc:240] Negotiation successful
0807 07:01:34.366072 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":358,"threads_started":1}
I20260807 07:01:34.390100 24700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:34.374609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:50659 (local address 127.21.83.62:34197)
0807 07:01:34.374786 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:34.374790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:34.375375 (+ 585us) server_negotiation.cc:407] Connection header received
0807 07:01:34.375523 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:34.375529 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:34.375627 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:34.375750 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:34.376928 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:34.378070 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:34.380021 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:34.380599 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:34.380741 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:34.383466 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:34.383471 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:34.383475 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:34.383514 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:34.386615 (+ 3101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:34.387736 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:34.387741 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:34.387743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:34.387941 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:34.388462 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:34.388467 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:34.388470 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:34.388875 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:34.389071 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:34.389486 (+ 415us) server_negotiation.cc:299] Negotiation successful
0807 07:01:34.389719 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 07:01:34.390210 24810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:34.374659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:34197 (local address 127.21.83.1:50659)
0807 07:01:34.375268 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:34.375284 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:34.375417 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:34.375797 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:34.375822 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:34.376348 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:01:34.376749 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:34.376762 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:34.378258 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:34.378265 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:34.379860 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:34.379876 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:34.381053 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:34.381061 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:34.381223 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:34.382042 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:01:34.382062 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:01:34.383243 (+ 1181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:01:34.386801 (+ 3558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:34.386807 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:34.386812 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:34.387498 (+ 686us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:34.388123 (+ 625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:34.388128 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:34.388132 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:34.388314 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:34.389025 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:01:34.389030 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:01:34.389232 (+ 202us) client_negotiation.cc:769] Sending connection context
0807 07:01:34.389361 (+ 129us) client_negotiation.cc:240] Negotiation successful
0807 07:01:34.389749 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":331,"threads_started":1}
I20260807 07:01:34.395713 24664 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.1:33999
I20260807 07:01:34.395756 24881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.1:33999 every 8 connection(s)
I20260807 07:01:34.397017 24664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
I20260807 07:01:34.399233 24882 heartbeater.cc:347] Connected to a master server at 127.21.83.62:34197
I20260807 07:01:34.399474 24882 heartbeater.cc:464] Registering TS with master...
I20260807 07:01:34.400195 24882 heartbeater.cc:511] Master 127.21.83.62:34197 requested a full tablet report, sending...
I20260807 07:01:34.402261 24560 ts_manager.cc:194] Registered new tserver with Master: 62aacfc55612476eac88f5a78966f57e (127.21.83.1:33999)
I20260807 07:01:34.403745 24560 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.1@KRBTEST.COM'} at 127.21.83.1:46389
I20260807 07:01:34.405491 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 24664
I20260807 07:01:34.405685 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-0/wal/instance
I20260807 07:01:34.406309 21836 external_mini_cluster.cc:1428] Setting key 53ac3a057716fbb0f0dccc8a17f7aab8
WARNING: no policy specified for kudu/127.21.83.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab.
Entry for principal kudu/127.21.83.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab.
I20260807 07:01:34.455405 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.2:0
--local_ip_for_outbound_sockets=127.21.83.2
--webserver_interface=127.21.83.2
--webserver_port=0
--tserver_master_addrs=127.21.83.62:34197
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab
--principal=kudu/127.21.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:42185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:01:34.791949 24892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:01:34.792281 24892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:01:34.792428 24892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:01:34.799917 24892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:01:34.800055 24892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:01:34.800173 24892 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 07:01:34.800246 24892 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 07:01:34.800354 24892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.2
I20260807 07:01:34.810474 24892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:42185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab
--principal=kudu/127.21.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.83.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.83.62:34197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.2
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:01:34.811868 24892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:01:34.813256 24892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:01:34.826524 24898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:34.826579 24901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:34.826613 24899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:01:34.827625 24892 server_base.cc:1034] running on GCE node
I20260807 07:01:34.828240 24892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:01:34.829247 24892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:01:34.830484 24892 hybrid_clock.cc:648] HybridClock initialized: now 1786086094830421 us; error 54 us; skew 500 ppm
I20260807 07:01:34.830955 24892 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:01:34 dist-test-slave-ncp9 krb5kdc[24491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086094, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:01:34.834296 24892 init.cc:377] Logged in from keytab as kudu/127.21.83.2@KRBTEST.COM (short username kudu)
I20260807 07:01:34.835888 24892 webserver.cc:460] Webserver started at http://127.21.83.2:43235/ using document root <none> and password file <none>
I20260807 07:01:34.836601 24892 fs_manager.cc:359] Metadata directory not provided
I20260807 07:01:34.836723 24892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:01:34.837090 24892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:01:34.840158 24892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-1/data/instance:
uuid: "633cabbcad0347b698839df57695c6cf"
format_stamp: "Formatted at 2026-08-07 07:01:34 on dist-test-slave-ncp9"
server_key: "1b2873916abbe2037e9408d1475bb395"
server_key_iv: "66c03e7a1f742f09ddf6890fa76e47af"
server_key_version: "encryptionkey@0"
I20260807 07:01:34.840988 24892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-1/wal/instance:
uuid: "633cabbcad0347b698839df57695c6cf"
format_stamp: "Formatted at 2026-08-07 07:01:34 on dist-test-slave-ncp9"
server_key: "1b2873916abbe2037e9408d1475bb395"
server_key_iv: "66c03e7a1f742f09ddf6890fa76e47af"
server_key_version: "encryptionkey@0"
I20260807 07:01:34.846436 24892 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 07:01:34.850149 24914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:34.851671 24892 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 07:01:34.851856 24892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-1/wal
uuid: "633cabbcad0347b698839df57695c6cf"
format_stamp: "Formatted at 2026-08-07 07:01:34 on dist-test-slave-ncp9"
server_key: "1b2873916abbe2037e9408d1475bb395"
server_key_iv: "66c03e7a1f742f09ddf6890fa76e47af"
server_key_version: "encryptionkey@0"
I20260807 07:01:34.852083 24892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:01:34.883615 24892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:01:34.888396 24892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:01:34.888752 24892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:01:34.891650 24892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:01:34.891768 24892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:34.891901 24892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:01:34.891978 24892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 07:01:34 dist-test-slave-ncp9 krb5kdc[24491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086094, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.2@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:01:34.916289 24929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:34.892411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:37603 (local address 127.21.83.62:34197)
0807 07:01:34.892974 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:34.892981 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:34.894351 (+ 1370us) server_negotiation.cc:407] Connection header received
0807 07:01:34.897125 (+ 2774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:34.897133 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:34.897238 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:34.897391 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:34.899705 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:34.900702 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:34.903102 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:34.903676 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:34.903819 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:34.908914 (+ 5095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:34.908923 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:34.908928 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:34.908975 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:34.912231 (+ 3256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:34.913343 (+ 1112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:34.913347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:34.913349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:34.913568 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:34.914103 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:34.914109 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:34.914112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:34.914431 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:34.914600 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:34.915812 (+ 1212us) server_negotiation.cc:299] Negotiation successful
0807 07:01:34.916006 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":287,"threads_started":1}
I20260807 07:01:34.918470 24930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:34.893147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:34197 (local address 127.21.83.2:37603)
0807 07:01:34.894152 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:34.894200 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:34.896703 (+ 2503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:34.897708 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:34.897722 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:34.898503 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:01:34.899479 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:34.899515 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:34.900874 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:34.900879 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:34.902897 (+ 2018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:34.902913 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:34.904141 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:34.904146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:34.904433 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:34.905289 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:01:34.905337 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:01:34.908571 (+ 3234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:01:34.912428 (+ 3857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:34.912438 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:34.912462 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:34.913170 (+ 708us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:34.913730 (+ 560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:34.913735 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:34.913739 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:34.913975 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:34.914595 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:01:34.914604 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:01:34.915216 (+ 612us) client_negotiation.cc:769] Sending connection context
0807 07:01:34.915679 (+ 463us) client_negotiation.cc:240] Negotiation successful
0807 07:01:34.916355 (+ 676us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":455,"threads_started":1}
I20260807 07:01:34.931370 24892 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.2:43979
I20260807 07:01:34.931470 25110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.2:43979 every 8 connection(s)
I20260807 07:01:34.932737 24892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
I20260807 07:01:34.935001 25113 heartbeater.cc:347] Connected to a master server at 127.21.83.62:34197
I20260807 07:01:34.935249 25113 heartbeater.cc:464] Registering TS with master...
I20260807 07:01:34.935913 25113 heartbeater.cc:511] Master 127.21.83.62:34197 requested a full tablet report, sending...
I20260807 07:01:34.937430 24560 ts_manager.cc:194] Registered new tserver with Master: 633cabbcad0347b698839df57695c6cf (127.21.83.2:43979)
I20260807 07:01:34.938608 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 24892
I20260807 07:01:34.938643 24560 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.2@KRBTEST.COM'} at 127.21.83.2:37603
I20260807 07:01:34.938808 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-1/wal/instance
I20260807 07:01:34.939390 21836 external_mini_cluster.cc:1428] Setting key 310259bb4091c82954be22fb6d7199bf
I20260807 07:01:34.941597 24929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:34.926318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:53147 (local address 127.21.83.62:34197)
0807 07:01:34.926506 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:34.926512 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:34.927184 (+ 672us) server_negotiation.cc:407] Connection header received
0807 07:01:34.927485 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:34.927489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:34.927562 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:34.927653 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:34.929330 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:34.930204 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:34.932103 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:34.932501 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:34.932618 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:34.935112 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:34.935119 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:34.935124 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:34.935168 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:34.938459 (+ 3291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:34.939357 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:34.939363 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:34.939366 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:34.939537 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:34.939909 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:34.939915 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:34.939919 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:34.940378 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:34.940535 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:34.940995 (+ 460us) server_negotiation.cc:299] Negotiation successful
0807 07:01:34.941241 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 07:01:34.941718 25090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:34.926315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:34197 (local address 127.21.83.2:53147)
0807 07:01:34.927071 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:34.927092 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:34.927323 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:34.927852 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:34.927857 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:34.928582 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:01:34.929161 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:34.929178 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:34.930405 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:34.930411 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:34.931957 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:34.931971 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:34.932693 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:34.932700 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:34.932970 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:34.933696 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:01:34.933717 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:01:34.934920 (+ 1203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:01:34.938628 (+ 3708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:34.938635 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:34.938639 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:34.939219 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:34.939653 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:34.939659 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:34.939663 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:34.939801 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:34.940507 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:01:34.940512 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:01:34.940727 (+ 215us) client_negotiation.cc:769] Sending connection context
0807 07:01:34.940853 (+ 126us) client_negotiation.cc:240] Negotiation successful
0807 07:01:34.941251 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"mutex_wait_us":4,"thread_start_us":409,"threads_started":1}
WARNING: no policy specified for kudu/127.21.83.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab.
Entry for principal kudu/127.21.83.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab.
I20260807 07:01:34.989974 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.3:0
--local_ip_for_outbound_sockets=127.21.83.3
--webserver_interface=127.21.83.3
--webserver_port=0
--tserver_master_addrs=127.21.83.62:34197
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab
--principal=kudu/127.21.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:42185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:01:35.334203 25118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:01:35.334513 25118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:01:35.334661 25118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:01:35.341874 25118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:01:35.341988 25118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:01:35.342115 25118 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 07:01:35.342180 25118 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 07:01:35.342301 25118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.3
I20260807 07:01:35.352347 25118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:42185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab
--principal=kudu/127.21.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.83.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.83.62:34197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.3
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:01:35.353785 25118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:01:35.355171 25118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:01:35.369053 25127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:35.369087 25126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:35.369019 25129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:01:35.369900 25118 server_base.cc:1034] running on GCE node
I20260807 07:01:35.370599 25118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:01:35.371634 25118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:01:35.372869 25118 hybrid_clock.cc:648] HybridClock initialized: now 1786086095372806 us; error 58 us; skew 500 ppm
I20260807 07:01:35.373283 25118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:01:35 dist-test-slave-ncp9 krb5kdc[24491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086095, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:01:35.381343 25118 init.cc:377] Logged in from keytab as kudu/127.21.83.3@KRBTEST.COM (short username kudu)
I20260807 07:01:35.383162 25118 webserver.cc:460] Webserver started at http://127.21.83.3:46203/ using document root <none> and password file <none>
I20260807 07:01:35.383905 25118 fs_manager.cc:359] Metadata directory not provided
I20260807 07:01:35.384044 25118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:01:35.384377 25118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:01:35.387686 25118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-2/data/instance:
uuid: "8b9c6ea9dc714845ad211e5fd0e146cb"
format_stamp: "Formatted at 2026-08-07 07:01:35 on dist-test-slave-ncp9"
server_key: "2ece03c00be16fbd391c0afd318b5576"
server_key_iv: "0806bdec45ee2f43703ad757bf2b1d80"
server_key_version: "encryptionkey@0"
I20260807 07:01:35.388655 25118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-2/wal/instance:
uuid: "8b9c6ea9dc714845ad211e5fd0e146cb"
format_stamp: "Formatted at 2026-08-07 07:01:35 on dist-test-slave-ncp9"
server_key: "2ece03c00be16fbd391c0afd318b5576"
server_key_iv: "0806bdec45ee2f43703ad757bf2b1d80"
server_key_version: "encryptionkey@0"
I20260807 07:01:35.394459 25118 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260807 07:01:35.398269 25139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:35.399775 25118 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 07:01:35.399945 25118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-2/wal
uuid: "8b9c6ea9dc714845ad211e5fd0e146cb"
format_stamp: "Formatted at 2026-08-07 07:01:35 on dist-test-slave-ncp9"
server_key: "2ece03c00be16fbd391c0afd318b5576"
server_key_iv: "0806bdec45ee2f43703ad757bf2b1d80"
server_key_version: "encryptionkey@0"
I20260807 07:01:35.400151 25118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:01:35.406723 24882 heartbeater.cc:503] Master 127.21.83.62:34197 was elected leader, sending a full tablet report...
I20260807 07:01:35.417461 25118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:01:35.422484 25118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:01:35.422780 25118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:01:35.425450 25118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:01:35.425568 25118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:35.425685 25118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:01:35.425761 25118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 07:01:35 dist-test-slave-ncp9 krb5kdc[24491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086095, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.3@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:01:35.454322 24929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:35.426082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:50669 (local address 127.21.83.62:34197)
0807 07:01:35.426292 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:35.426298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:35.427810 (+ 1512us) server_negotiation.cc:407] Connection header received
0807 07:01:35.430156 (+ 2346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:35.430165 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:35.430296 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:35.430424 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:35.432181 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:35.433418 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:35.436134 (+ 2716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:35.436856 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:35.437038 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:35.443105 (+ 6067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:35.443116 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:35.443122 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:35.443183 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:35.448264 (+ 5081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:35.451052 (+ 2788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:35.451060 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:35.451064 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:35.451232 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:35.451863 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:35.451869 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:35.451872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:35.452313 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:35.452425 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:35.453651 (+ 1226us) server_negotiation.cc:299] Negotiation successful
0807 07:01:35.453932 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260807 07:01:35.456964 25157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:35.426673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:34197 (local address 127.21.83.3:50669)
0807 07:01:35.427592 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:35.427653 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:35.429696 (+ 2043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:35.430723 (+ 1027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:35.430733 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:35.431303 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:01:35.431955 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:35.431974 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:35.433624 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:35.433631 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:35.435914 (+ 2283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:35.435942 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:35.437195 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:35.437204 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:35.437459 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:35.438356 (+ 897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:01:35.438395 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:01:35.442633 (+ 4238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:01:35.450008 (+ 7375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:35.450021 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:35.450046 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:35.450887 (+ 841us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:35.451372 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:35.451378 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:35.451382 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:35.451693 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:35.452523 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:01:35.452530 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:01:35.453094 (+ 564us) client_negotiation.cc:769] Sending connection context
0807 07:01:35.454676 (+ 1582us) client_negotiation.cc:240] Negotiation successful
0807 07:01:35.455363 (+ 687us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":398,"threads_started":1}
I20260807 07:01:35.474117 25118 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.3:39465
I20260807 07:01:35.474169 25324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.3:39465 every 8 connection(s)
I20260807 07:01:35.475512 25118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
I20260807 07:01:35.477979 25325 heartbeater.cc:347] Connected to a master server at 127.21.83.62:34197
I20260807 07:01:35.478222 25325 heartbeater.cc:464] Registering TS with master...
I20260807 07:01:35.479143 25325 heartbeater.cc:511] Master 127.21.83.62:34197 requested a full tablet report, sending...
I20260807 07:01:35.480305 24929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:35.464962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:53963 (local address 127.21.83.62:34197)
0807 07:01:35.465170 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:35.465176 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:35.465917 (+ 741us) server_negotiation.cc:407] Connection header received
0807 07:01:35.466070 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:35.466076 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:35.466183 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:35.466310 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:35.467771 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:35.468985 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:35.470985 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:35.471629 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:35.471765 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:35.474433 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:35.474440 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:35.474445 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:35.474506 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:35.477519 (+ 3013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:35.478278 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:35.478284 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:35.478287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:35.478484 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:35.478854 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:35.478858 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:35.478861 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:35.479266 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:35.479423 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:35.479782 (+ 359us) server_negotiation.cc:299] Negotiation successful
0807 07:01:35.480034 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 07:01:35.480358 25290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:35.465021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:34197 (local address 127.21.83.3:53963)
0807 07:01:35.465789 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:35.465809 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:35.465949 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:35.466329 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:35.466332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:35.466993 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:01:35.467530 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:35.467547 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:35.469131 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:35.469136 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:35.470800 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:35.470824 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:35.472049 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:35.472055 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:35.472221 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:35.472995 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:01:35.473018 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:01:35.474225 (+ 1207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:01:35.477652 (+ 3427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:35.477657 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:35.477661 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:35.478153 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:35.478600 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:35.478603 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:35.478606 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:35.478740 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:35.479373 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:01:35.479377 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:01:35.479535 (+ 158us) client_negotiation.cc:769] Sending connection context
0807 07:01:35.479649 (+ 114us) client_negotiation.cc:240] Negotiation successful
0807 07:01:35.479912 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":452,"threads_started":1}
I20260807 07:01:35.480881 24559 ts_manager.cc:194] Registered new tserver with Master: 8b9c6ea9dc714845ad211e5fd0e146cb (127.21.83.3:39465)
I20260807 07:01:35.481953 24559 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.3@KRBTEST.COM'} at 127.21.83.3:50669
I20260807 07:01:35.484663 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 25118
I20260807 07:01:35.484824 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-2/wal/instance
I20260807 07:01:35.485304 21836 external_mini_cluster.cc:1428] Setting key 04e429ea21cb4597133620d71ba17f5c
I20260807 07:01:35.488261 21836 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 07:01:35 dist-test-slave-ncp9 krb5kdc[24491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 07:01:35 dist-test-slave-ncp9 krb5kdc[24491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:01:35.543051 24929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:35.528729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37284 (local address 127.21.83.62:34197)
0807 07:01:35.528969 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:35.528976 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:35.529231 (+ 255us) server_negotiation.cc:407] Connection header received
0807 07:01:35.529426 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:35.529430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:35.529503 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:35.529600 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:35.531045 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:35.531789 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:35.533052 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:35.533484 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:35.533611 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:35.537587 (+ 3976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:35.537594 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:35.537597 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:35.537644 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:35.540148 (+ 2504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:35.540928 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:35.540934 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:35.540937 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:35.541094 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:35.541505 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:35.541508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:35.541511 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:35.541830 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:35.542093 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:35.542457 (+ 364us) server_negotiation.cc:299] Negotiation successful
0807 07:01:35.542720 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 07:01:35.547473 24559 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37284:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 07:01:35.554535 24559 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 07:01:35.579430 25344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:35.567371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35030 (local address 127.21.83.1:33999)
0807 07:01:35.568495 (+ 1124us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:35.568504 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:35.569789 (+ 1285us) server_negotiation.cc:407] Connection header received
0807 07:01:35.570144 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:35.570153 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:35.570410 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:35.570579 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:01:35.571547 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:35.573039 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:35.575918 (+ 2879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:35.577778 (+ 1860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:35.577907 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:35.578145 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:35.578909 (+ 764us) server_negotiation.cc:299] Negotiation successful
0807 07:01:35.579109 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":970,"thread_start_us":825,"threads_started":1}
I20260807 07:01:35.580696 25343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:35.567242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:33999 (local address 127.0.0.1:35030)
0807 07:01:35.569605 (+ 2363us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:35.569670 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:35.569901 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:35.570891 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:35.570898 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:35.570927 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:01:35.571348 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:35.571363 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:35.573247 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:35.573253 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:35.575678 (+ 2425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:35.575695 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:35.578346 (+ 2651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:35.578353 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:35.578606 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:35.578674 (+ 68us) client_negotiation.cc:769] Sending connection context
0807 07:01:35.580011 (+ 1337us) client_negotiation.cc:240] Negotiation successful
0807 07:01:35.580249 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2089,"spinlock_wait_cycles":4480,"thread_start_us":1896,"threads_started":1}
I20260807 07:01:35.582425 25341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:35.567302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.2:43979 (local address 127.0.0.1:42646)
0807 07:01:35.569605 (+ 2303us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:35.569729 (+ 124us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:35.569901 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:35.572063 (+ 2162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:35.572067 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:35.572092 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:01:35.572505 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:35.572518 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:35.575784 (+ 3266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:35.575790 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:35.578051 (+ 2261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:35.578069 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:35.581617 (+ 3548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:35.581624 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:35.581798 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:35.581823 (+ 25us) client_negotiation.cc:769] Sending connection context
0807 07:01:35.581903 (+ 80us) client_negotiation.cc:240] Negotiation successful
0807 07:01:35.582045 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2075,"spinlock_wait_cycles":10112,"thread_start_us":1849,"threads_started":1}
I20260807 07:01:35.582477 25345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:35.567888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42646 (local address 127.21.83.2:43979)
0807 07:01:35.568941 (+ 1053us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:35.568949 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:35.571294 (+ 2345us) server_negotiation.cc:407] Connection header received
0807 07:01:35.571436 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:35.571444 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:35.571668 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:35.571864 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:01:35.572713 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:35.575600 (+ 2887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:35.579848 (+ 4248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:35.581345 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:35.581472 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:35.581691 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:35.581998 (+ 307us) server_negotiation.cc:299] Negotiation successful
0807 07:01:35.582207 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":868,"thread_start_us":710,"threads_started":1}
I20260807 07:01:35.587149 24766 tablet_service.cc:1467] Processing CreateTablet for tablet c071b3f1a36d46f4b02024cba222f54b (DEFAULT_TABLE table=test-table [id=bb8976c963184c3b8e767db911321112]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:01:35.588387 24766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c071b3f1a36d46f4b02024cba222f54b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:01:35.591231 25000 tablet_service.cc:1467] Processing CreateTablet for tablet c071b3f1a36d46f4b02024cba222f54b (DEFAULT_TABLE table=test-table [id=bb8976c963184c3b8e767db911321112]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:01:35.592613 25000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c071b3f1a36d46f4b02024cba222f54b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:01:35.595728 25343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:35.569993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.3:39465 (local address 127.0.0.1:47534)
0807 07:01:35.581232 (+ 11239us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:35.581253 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:35.581371 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:35.589989 (+ 8618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:35.589994 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:35.590023 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:01:35.590226 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:35.590239 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:35.592454 (+ 2215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:35.592460 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:35.593739 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:35.593753 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:35.594976 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:35.594982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:35.595172 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:35.595206 (+ 34us) client_negotiation.cc:769] Sending connection context
0807 07:01:35.595289 (+ 83us) client_negotiation.cc:240] Negotiation successful
0807 07:01:35.595436 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11134,"thread_start_us":12784,"threads_started":1}
I20260807 07:01:35.595911 25349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:35.570382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47534 (local address 127.21.83.3:39465)
0807 07:01:35.589497 (+ 19115us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:35.589503 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:35.589533 (+ 30us) server_negotiation.cc:407] Connection header received
0807 07:01:35.589617 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:35.589622 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:35.589792 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:35.589977 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:01:35.590400 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:35.592310 (+ 1910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:35.593896 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:35.594810 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:35.594927 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:35.595168 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:35.595364 (+ 196us) server_negotiation.cc:299] Negotiation successful
0807 07:01:35.595509 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18948,"thread_start_us":18746,"threads_started":1}
I20260807 07:01:35.599069 25210 tablet_service.cc:1467] Processing CreateTablet for tablet c071b3f1a36d46f4b02024cba222f54b (DEFAULT_TABLE table=test-table [id=bb8976c963184c3b8e767db911321112]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:01:35.600369 25210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c071b3f1a36d46f4b02024cba222f54b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:01:35.603073 25350 tablet_bootstrap.cc:492] T c071b3f1a36d46f4b02024cba222f54b P 62aacfc55612476eac88f5a78966f57e: Bootstrap starting.
I20260807 07:01:35.606419 25352 tablet_bootstrap.cc:492] T c071b3f1a36d46f4b02024cba222f54b P 633cabbcad0347b698839df57695c6cf: Bootstrap starting.
I20260807 07:01:35.606858 25350 tablet_bootstrap.cc:654] T c071b3f1a36d46f4b02024cba222f54b P 62aacfc55612476eac88f5a78966f57e: Neither blocks nor log segments found. Creating new log.
I20260807 07:01:35.608220 25350 log.cc:824] T c071b3f1a36d46f4b02024cba222f54b P 62aacfc55612476eac88f5a78966f57e: Log is configured to *not* fsync() on all Append() calls
I20260807 07:01:35.609696 25352 tablet_bootstrap.cc:654] T c071b3f1a36d46f4b02024cba222f54b P 633cabbcad0347b698839df57695c6cf: Neither blocks nor log segments found. Creating new log.
I20260807 07:01:35.610744 25352 log.cc:824] T c071b3f1a36d46f4b02024cba222f54b P 633cabbcad0347b698839df57695c6cf: Log is configured to *not* fsync() on all Append() calls
I20260807 07:01:35.612025 25350 tablet_bootstrap.cc:492] T c071b3f1a36d46f4b02024cba222f54b P 62aacfc55612476eac88f5a78966f57e: No bootstrap required, opened a new log
I20260807 07:01:35.612326 25350 ts_tablet_manager.cc:1397] T c071b3f1a36d46f4b02024cba222f54b P 62aacfc55612476eac88f5a78966f57e: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260807 07:01:35.613153 25352 tablet_bootstrap.cc:492] T c071b3f1a36d46f4b02024cba222f54b P 633cabbcad0347b698839df57695c6cf: No bootstrap required, opened a new log
I20260807 07:01:35.613389 25352 ts_tablet_manager.cc:1397] T c071b3f1a36d46f4b02024cba222f54b P 633cabbcad0347b698839df57695c6cf: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260807 07:01:35.614423 25358 tablet_bootstrap.cc:492] T c071b3f1a36d46f4b02024cba222f54b P 8b9c6ea9dc714845ad211e5fd0e146cb: Bootstrap starting.
I20260807 07:01:35.618279 25358 tablet_bootstrap.cc:654] T c071b3f1a36d46f4b02024cba222f54b P 8b9c6ea9dc714845ad211e5fd0e146cb: Neither blocks nor log segments found. Creating new log.
I20260807 07:01:35.620747 25358 log.cc:824] T c071b3f1a36d46f4b02024cba222f54b P 8b9c6ea9dc714845ad211e5fd0e146cb: Log is configured to *not* fsync() on all Append() calls
I20260807 07:01:35.620822 25350 raft_consensus.cc:348] T c071b3f1a36d46f4b02024cba222f54b P 62aacfc55612476eac88f5a78966f57e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } }
I20260807 07:01:35.621287 25350 raft_consensus.cc:374] T c071b3f1a36d46f4b02024cba222f54b P 62aacfc55612476eac88f5a78966f57e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:01:35.621400 25350 raft_consensus.cc:729] T c071b3f1a36d46f4b02024cba222f54b P 62aacfc55612476eac88f5a78966f57e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62aacfc55612476eac88f5a78966f57e, State: Initialized, Role: FOLLOWER
I20260807 07:01:35.622164 25350 consensus_queue.cc:260] T c071b3f1a36d46f4b02024cba222f54b P 62aacfc55612476eac88f5a78966f57e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } }
I20260807 07:01:35.623699 25350 ts_tablet_manager.cc:1428] T c071b3f1a36d46f4b02024cba222f54b P 62aacfc55612476eac88f5a78966f57e: Time spent starting tablet: real 0.011s user 0.007s sys 0.005s
I20260807 07:01:35.625190 25352 raft_consensus.cc:348] T c071b3f1a36d46f4b02024cba222f54b P 633cabbcad0347b698839df57695c6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } }
I20260807 07:01:35.625622 25352 raft_consensus.cc:374] T c071b3f1a36d46f4b02024cba222f54b P 633cabbcad0347b698839df57695c6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:01:35.625732 25352 raft_consensus.cc:729] T c071b3f1a36d46f4b02024cba222f54b P 633cabbcad0347b698839df57695c6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 633cabbcad0347b698839df57695c6cf, State: Initialized, Role: FOLLOWER
I20260807 07:01:35.626363 25352 consensus_queue.cc:260] T c071b3f1a36d46f4b02024cba222f54b P 633cabbcad0347b698839df57695c6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } }
I20260807 07:01:35.627548 25358 tablet_bootstrap.cc:492] T c071b3f1a36d46f4b02024cba222f54b P 8b9c6ea9dc714845ad211e5fd0e146cb: No bootstrap required, opened a new log
I20260807 07:01:35.627530 25113 heartbeater.cc:503] Master 127.21.83.62:34197 was elected leader, sending a full tablet report...
I20260807 07:01:35.627863 25358 ts_tablet_manager.cc:1397] T c071b3f1a36d46f4b02024cba222f54b P 8b9c6ea9dc714845ad211e5fd0e146cb: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260807 07:01:35.628216 25352 ts_tablet_manager.cc:1428] T c071b3f1a36d46f4b02024cba222f54b P 633cabbcad0347b698839df57695c6cf: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260807 07:01:35.636240 25358 raft_consensus.cc:348] T c071b3f1a36d46f4b02024cba222f54b P 8b9c6ea9dc714845ad211e5fd0e146cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } }
I20260807 07:01:35.636639 25358 raft_consensus.cc:374] T c071b3f1a36d46f4b02024cba222f54b P 8b9c6ea9dc714845ad211e5fd0e146cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:01:35.636757 25358 raft_consensus.cc:729] T c071b3f1a36d46f4b02024cba222f54b P 8b9c6ea9dc714845ad211e5fd0e146cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b9c6ea9dc714845ad211e5fd0e146cb, State: Initialized, Role: FOLLOWER
I20260807 07:01:35.637404 25358 consensus_queue.cc:260] T c071b3f1a36d46f4b02024cba222f54b P 8b9c6ea9dc714845ad211e5fd0e146cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } }
I20260807 07:01:35.638849 25325 heartbeater.cc:503] Master 127.21.83.62:34197 was elected leader, sending a full tablet report...
I20260807 07:01:35.639334 25358 ts_tablet_manager.cc:1428] T c071b3f1a36d46f4b02024cba222f54b P 8b9c6ea9dc714845ad211e5fd0e146cb: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260807 07:01:35.641681 25365 raft_consensus.cc:482] T c071b3f1a36d46f4b02024cba222f54b P 8b9c6ea9dc714845ad211e5fd0e146cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:01:35.641963 25365 raft_consensus.cc:504] T c071b3f1a36d46f4b02024cba222f54b P 8b9c6ea9dc714845ad211e5fd0e146cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } }
I20260807 07:01:35.643786 25365 leader_election.cc:288] T c071b3f1a36d46f4b02024cba222f54b P 8b9c6ea9dc714845ad211e5fd0e146cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62aacfc55612476eac88f5a78966f57e (127.21.83.1:33999), 633cabbcad0347b698839df57695c6cf (127.21.83.2:43979)
W20260807 07:01:35.649432 24885 tablet.cc:2369] T c071b3f1a36d46f4b02024cba222f54b P 62aacfc55612476eac88f5a78966f57e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 07:01:35.650765 25157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:35.644144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.2:43979 (local address 127.21.83.3:60741)
0807 07:01:35.644378 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:35.644397 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:35.644515 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:35.645364 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:35.645369 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:35.645399 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:01:35.645606 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:35.645618 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:35.646847 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:35.646853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:35.648509 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:35.648526 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:35.650103 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:35.650109 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:35.650305 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:35.650343 (+ 38us) client_negotiation.cc:769] Sending connection context
0807 07:01:35.650427 (+ 84us) client_negotiation.cc:240] Negotiation successful
0807 07:01:35.650535 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":3,"spinlock_wait_cycles":896}
I20260807 07:01:35.650887 25345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:35.644760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:60741 (local address 127.21.83.2:43979)
0807 07:01:35.645049 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:35.645055 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:35.645075 (+ 20us) server_negotiation.cc:407] Connection header received
0807 07:01:35.645152 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:35.645156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:35.645234 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:35.645333 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:01:35.645748 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:35.646699 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:35.648700 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:35.649915 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:35.650051 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:35.650164 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:35.650476 (+ 312us) server_negotiation.cc:299] Negotiation successful
0807 07:01:35.650617 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 07:01:35.651670 25344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:35.644323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:51715 (local address 127.21.83.1:33999)
0807 07:01:35.644508 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:35.644513 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:35.645413 (+ 900us) server_negotiation.cc:407] Connection header received
0807 07:01:35.645596 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:35.645601 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:35.645692 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:35.645776 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:01:35.646394 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:35.647289 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:35.649392 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:35.650579 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:35.650688 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:35.650788 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:35.651351 (+ 563us) server_negotiation.cc:299] Negotiation successful
0807 07:01:35.651491 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 07:01:35.651770 25366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:35.644146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:33999 (local address 127.21.83.3:51715)
0807 07:01:35.645309 (+ 1163us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:35.645326 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:35.645441 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:35.645806 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:35.645810 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:35.645835 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:01:35.646229 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:35.646241 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:35.647440 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:35.647444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:35.649200 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:35.649216 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:35.650995 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:35.651001 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:35.651178 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:35.651212 (+ 34us) client_negotiation.cc:769] Sending connection context
0807 07:01:35.651290 (+ 78us) client_negotiation.cc:240] Negotiation successful
0807 07:01:35.651420 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1024,"mutex_wait_us":423,"thread_start_us":415,"threads_started":1}
I20260807 07:01:35.651947 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c071b3f1a36d46f4b02024cba222f54b" candidate_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "633cabbcad0347b698839df57695c6cf" is_pre_election: true
I20260807 07:01:35.652511 25031 raft_consensus.cc:2454] T c071b3f1a36d46f4b02024cba222f54b P 633cabbcad0347b698839df57695c6cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b9c6ea9dc714845ad211e5fd0e146cb in term 0.
I20260807 07:01:35.652729 24807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c071b3f1a36d46f4b02024cba222f54b" candidate_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62aacfc55612476eac88f5a78966f57e" is_pre_election: true
I20260807 07:01:35.653194 25148 leader_election.cc:302] T c071b3f1a36d46f4b02024cba222f54b P 8b9c6ea9dc714845ad211e5fd0e146cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 633cabbcad0347b698839df57695c6cf, 8b9c6ea9dc714845ad211e5fd0e146cb; no voters:
I20260807 07:01:35.653244 24807 raft_consensus.cc:2454] T c071b3f1a36d46f4b02024cba222f54b P 62aacfc55612476eac88f5a78966f57e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b9c6ea9dc714845ad211e5fd0e146cb in term 0.
I20260807 07:01:35.653581 25365 raft_consensus.cc:2781] T c071b3f1a36d46f4b02024cba222f54b P 8b9c6ea9dc714845ad211e5fd0e146cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 07:01:35.653752 25365 raft_consensus.cc:482] T c071b3f1a36d46f4b02024cba222f54b P 8b9c6ea9dc714845ad211e5fd0e146cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:01:35.653869 25365 raft_consensus.cc:3037] T c071b3f1a36d46f4b02024cba222f54b P 8b9c6ea9dc714845ad211e5fd0e146cb [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:01:35.656253 25365 raft_consensus.cc:504] T c071b3f1a36d46f4b02024cba222f54b P 8b9c6ea9dc714845ad211e5fd0e146cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } }
I20260807 07:01:35.656968 25365 leader_election.cc:288] T c071b3f1a36d46f4b02024cba222f54b P 8b9c6ea9dc714845ad211e5fd0e146cb [CANDIDATE]: Term 1 election: Requested vote from peers 62aacfc55612476eac88f5a78966f57e (127.21.83.1:33999), 633cabbcad0347b698839df57695c6cf (127.21.83.2:43979)
I20260807 07:01:35.657483 24807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c071b3f1a36d46f4b02024cba222f54b" candidate_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62aacfc55612476eac88f5a78966f57e"
I20260807 07:01:35.657545 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c071b3f1a36d46f4b02024cba222f54b" candidate_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "633cabbcad0347b698839df57695c6cf"
I20260807 07:01:35.657694 24807 raft_consensus.cc:3037] T c071b3f1a36d46f4b02024cba222f54b P 62aacfc55612476eac88f5a78966f57e [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:01:35.657759 25031 raft_consensus.cc:3037] T c071b3f1a36d46f4b02024cba222f54b P 633cabbcad0347b698839df57695c6cf [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:01:35.660104 24807 raft_consensus.cc:2454] T c071b3f1a36d46f4b02024cba222f54b P 62aacfc55612476eac88f5a78966f57e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b9c6ea9dc714845ad211e5fd0e146cb in term 1.
I20260807 07:01:35.660189 25031 raft_consensus.cc:2454] T c071b3f1a36d46f4b02024cba222f54b P 633cabbcad0347b698839df57695c6cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b9c6ea9dc714845ad211e5fd0e146cb in term 1.
I20260807 07:01:35.660589 25140 leader_election.cc:302] T c071b3f1a36d46f4b02024cba222f54b P 8b9c6ea9dc714845ad211e5fd0e146cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62aacfc55612476eac88f5a78966f57e, 8b9c6ea9dc714845ad211e5fd0e146cb; no voters:
I20260807 07:01:35.660938 25365 raft_consensus.cc:2781] T c071b3f1a36d46f4b02024cba222f54b P 8b9c6ea9dc714845ad211e5fd0e146cb [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:01:35.661396 25365 raft_consensus.cc:686] T c071b3f1a36d46f4b02024cba222f54b P 8b9c6ea9dc714845ad211e5fd0e146cb [term 1 LEADER]: Becoming Leader. State: Replica: 8b9c6ea9dc714845ad211e5fd0e146cb, State: Running, Role: LEADER
I20260807 07:01:35.661900 25365 consensus_queue.cc:237] T c071b3f1a36d46f4b02024cba222f54b P 8b9c6ea9dc714845ad211e5fd0e146cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } }
I20260807 07:01:35.666895 24560 catalog_manager.cc:5495] T c071b3f1a36d46f4b02024cba222f54b P 8b9c6ea9dc714845ad211e5fd0e146cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b9c6ea9dc714845ad211e5fd0e146cb (127.21.83.3). New cstate: current_term: 1 leader_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } health_report { overall_health: HEALTHY } } }
W20260807 07:01:35.685166 25114 tablet.cc:2369] T c071b3f1a36d46f4b02024cba222f54b P 633cabbcad0347b698839df57695c6cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 07 07:01:35 dist-test-slave-ncp9 krb5kdc[24491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086093, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
W20260807 07:01:35.727427 25326 tablet.cc:2369] T c071b3f1a36d46f4b02024cba222f54b P 8b9c6ea9dc714845ad211e5fd0e146cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 07:01:35.733877 24929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:35.719580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37300 (local address 127.21.83.62:34197)
0807 07:01:35.719893 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:35.719900 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:35.720386 (+ 486us) server_negotiation.cc:407] Connection header received
0807 07:01:35.720564 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:35.720569 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:35.720628 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:35.720710 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:35.722014 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:35.722773 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:35.724026 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:35.724431 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:35.724540 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:35.728407 (+ 3867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:35.728412 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:35.728416 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:35.728452 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:35.730926 (+ 2474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:35.731822 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:35.731826 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:35.731828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:35.731982 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:35.732354 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:35.732357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:35.732359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:35.732624 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:35.732759 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:35.733239 (+ 480us) server_negotiation.cc:299] Negotiation successful
0807 07:01:35.733588 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":140,"server-negotiator.queue_time_us":139}
I20260807 07:01:35.733942 25379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:35.719580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:34197 (local address 127.0.0.1:37300)
0807 07:01:35.720274 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:35.720291 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:35.720450 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:35.720758 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:35.720763 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:35.721599 (+ 836us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:01:35.721870 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:35.721880 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:35.722911 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:35.722915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:35.723890 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:35.723903 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:35.724618 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:35.724624 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:35.724781 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:35.725508 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:01:35.725526 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:01:35.728157 (+ 2631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:01:35.731088 (+ 2931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:35.731105 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:35.731113 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:35.731714 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:35.732077 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:35.732082 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:35.732086 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:35.732257 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:35.732728 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:01:35.732740 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:01:35.732961 (+ 221us) client_negotiation.cc:769] Sending connection context
0807 07:01:35.733139 (+ 178us) client_negotiation.cc:240] Negotiation successful
0807 07:01:35.733454 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":407,"threads_started":1}
I20260807 07:01:35.739574 24560 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:37300:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260807 07:01:35.740980 24560 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 07:01:35.748507 24766 tablet_service.cc:1467] Processing CreateTablet for tablet 64e108423d4f4cd5addf58e34d573c19 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=928d167f2c2d44f19611aadd28a6113c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 07:01:35.748596 25000 tablet_service.cc:1467] Processing CreateTablet for tablet 64e108423d4f4cd5addf58e34d573c19 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=928d167f2c2d44f19611aadd28a6113c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 07:01:35.748507 25210 tablet_service.cc:1467] Processing CreateTablet for tablet 64e108423d4f4cd5addf58e34d573c19 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=928d167f2c2d44f19611aadd28a6113c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 07:01:35.749570 25000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64e108423d4f4cd5addf58e34d573c19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:01:35.749574 25210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64e108423d4f4cd5addf58e34d573c19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:01:35.749575 24766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64e108423d4f4cd5addf58e34d573c19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:01:35.755863 25352 tablet_bootstrap.cc:492] T 64e108423d4f4cd5addf58e34d573c19 P 633cabbcad0347b698839df57695c6cf: Bootstrap starting.
I20260807 07:01:35.756681 25358 tablet_bootstrap.cc:492] T 64e108423d4f4cd5addf58e34d573c19 P 8b9c6ea9dc714845ad211e5fd0e146cb: Bootstrap starting.
I20260807 07:01:35.757581 25352 tablet_bootstrap.cc:654] T 64e108423d4f4cd5addf58e34d573c19 P 633cabbcad0347b698839df57695c6cf: Neither blocks nor log segments found. Creating new log.
I20260807 07:01:35.757949 25358 tablet_bootstrap.cc:654] T 64e108423d4f4cd5addf58e34d573c19 P 8b9c6ea9dc714845ad211e5fd0e146cb: Neither blocks nor log segments found. Creating new log.
I20260807 07:01:35.759311 25350 tablet_bootstrap.cc:492] T 64e108423d4f4cd5addf58e34d573c19 P 62aacfc55612476eac88f5a78966f57e: Bootstrap starting.
I20260807 07:01:35.759770 25352 tablet_bootstrap.cc:492] T 64e108423d4f4cd5addf58e34d573c19 P 633cabbcad0347b698839df57695c6cf: No bootstrap required, opened a new log
I20260807 07:01:35.759912 25352 ts_tablet_manager.cc:1397] T 64e108423d4f4cd5addf58e34d573c19 P 633cabbcad0347b698839df57695c6cf: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260807 07:01:35.760607 25358 tablet_bootstrap.cc:492] T 64e108423d4f4cd5addf58e34d573c19 P 8b9c6ea9dc714845ad211e5fd0e146cb: No bootstrap required, opened a new log
I20260807 07:01:35.760730 25350 tablet_bootstrap.cc:654] T 64e108423d4f4cd5addf58e34d573c19 P 62aacfc55612476eac88f5a78966f57e: Neither blocks nor log segments found. Creating new log.
I20260807 07:01:35.760758 25358 ts_tablet_manager.cc:1397] T 64e108423d4f4cd5addf58e34d573c19 P 8b9c6ea9dc714845ad211e5fd0e146cb: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260807 07:01:35.760782 25352 raft_consensus.cc:348] T 64e108423d4f4cd5addf58e34d573c19 P 633cabbcad0347b698839df57695c6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } }
I20260807 07:01:35.761076 25352 raft_consensus.cc:374] T 64e108423d4f4cd5addf58e34d573c19 P 633cabbcad0347b698839df57695c6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:01:35.761185 25352 raft_consensus.cc:729] T 64e108423d4f4cd5addf58e34d573c19 P 633cabbcad0347b698839df57695c6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 633cabbcad0347b698839df57695c6cf, State: Initialized, Role: FOLLOWER
I20260807 07:01:35.761423 25352 consensus_queue.cc:260] T 64e108423d4f4cd5addf58e34d573c19 P 633cabbcad0347b698839df57695c6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } }
I20260807 07:01:35.761564 25358 raft_consensus.cc:348] T 64e108423d4f4cd5addf58e34d573c19 P 8b9c6ea9dc714845ad211e5fd0e146cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } }
I20260807 07:01:35.761788 25358 raft_consensus.cc:374] T 64e108423d4f4cd5addf58e34d573c19 P 8b9c6ea9dc714845ad211e5fd0e146cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:01:35.761878 25358 raft_consensus.cc:729] T 64e108423d4f4cd5addf58e34d573c19 P 8b9c6ea9dc714845ad211e5fd0e146cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b9c6ea9dc714845ad211e5fd0e146cb, State: Initialized, Role: FOLLOWER
I20260807 07:01:35.762229 25352 ts_tablet_manager.cc:1428] T 64e108423d4f4cd5addf58e34d573c19 P 633cabbcad0347b698839df57695c6cf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260807 07:01:35.762094 25358 consensus_queue.cc:260] T 64e108423d4f4cd5addf58e34d573c19 P 8b9c6ea9dc714845ad211e5fd0e146cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } }
I20260807 07:01:35.761976 25363 tablet_replica.cc:439] T 64e108423d4f4cd5addf58e34d573c19 P 633cabbcad0347b698839df57695c6cf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } } }
I20260807 07:01:35.762439 25363 tablet_replica.cc:442] T 64e108423d4f4cd5addf58e34d573c19 P 633cabbcad0347b698839df57695c6cf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 07:01:35.762514 25365 tablet_replica.cc:439] T 64e108423d4f4cd5addf58e34d573c19 P 8b9c6ea9dc714845ad211e5fd0e146cb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } } }
I20260807 07:01:35.762729 25358 ts_tablet_manager.cc:1428] T 64e108423d4f4cd5addf58e34d573c19 P 8b9c6ea9dc714845ad211e5fd0e146cb: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260807 07:01:35.762792 25365 tablet_replica.cc:442] T 64e108423d4f4cd5addf58e34d573c19 P 8b9c6ea9dc714845ad211e5fd0e146cb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 07:01:35.765069 25350 tablet_bootstrap.cc:492] T 64e108423d4f4cd5addf58e34d573c19 P 62aacfc55612476eac88f5a78966f57e: No bootstrap required, opened a new log
I20260807 07:01:35.765206 25350 ts_tablet_manager.cc:1397] T 64e108423d4f4cd5addf58e34d573c19 P 62aacfc55612476eac88f5a78966f57e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260807 07:01:35.765921 25350 raft_consensus.cc:348] T 64e108423d4f4cd5addf58e34d573c19 P 62aacfc55612476eac88f5a78966f57e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } }
I20260807 07:01:35.766182 25350 raft_consensus.cc:374] T 64e108423d4f4cd5addf58e34d573c19 P 62aacfc55612476eac88f5a78966f57e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:01:35.766289 25350 raft_consensus.cc:729] T 64e108423d4f4cd5addf58e34d573c19 P 62aacfc55612476eac88f5a78966f57e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62aacfc55612476eac88f5a78966f57e, State: Initialized, Role: FOLLOWER
I20260807 07:01:35.766530 25350 consensus_queue.cc:260] T 64e108423d4f4cd5addf58e34d573c19 P 62aacfc55612476eac88f5a78966f57e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } }
I20260807 07:01:35.766978 25361 tablet_replica.cc:439] T 64e108423d4f4cd5addf58e34d573c19 P 62aacfc55612476eac88f5a78966f57e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } } }
I20260807 07:01:35.767202 25350 ts_tablet_manager.cc:1428] T 64e108423d4f4cd5addf58e34d573c19 P 62aacfc55612476eac88f5a78966f57e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260807 07:01:35.767334 25361 tablet_replica.cc:442] T 64e108423d4f4cd5addf58e34d573c19 P 62aacfc55612476eac88f5a78966f57e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 07:01:35.869328 25363 raft_consensus.cc:482] T 64e108423d4f4cd5addf58e34d573c19 P 633cabbcad0347b698839df57695c6cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:01:35.869756 25363 raft_consensus.cc:504] T 64e108423d4f4cd5addf58e34d573c19 P 633cabbcad0347b698839df57695c6cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } }
I20260807 07:01:35.871371 25363 leader_election.cc:288] T 64e108423d4f4cd5addf58e34d573c19 P 633cabbcad0347b698839df57695c6cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62aacfc55612476eac88f5a78966f57e (127.21.83.1:33999), 8b9c6ea9dc714845ad211e5fd0e146cb (127.21.83.3:39465)
I20260807 07:01:35.879805 25344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:35.871855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:33363 (local address 127.21.83.1:33999)
0807 07:01:35.872057 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:35.872061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:35.872547 (+ 486us) server_negotiation.cc:407] Connection header received
0807 07:01:35.872834 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:35.872840 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:35.872949 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:35.873048 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:01:35.873740 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:35.874723 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:35.877369 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:35.878587 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:35.878740 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:35.878853 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:35.879349 (+ 496us) server_negotiation.cc:299] Negotiation successful
0807 07:01:35.879498 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 07:01:35.879863 25389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:35.871780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:33999 (local address 127.21.83.2:33363)
0807 07:01:35.872455 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:35.872473 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:35.872630 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:35.873199 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:35.873204 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:35.873231 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:01:35.873576 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:35.873586 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:35.874873 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:35.874877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:35.877169 (+ 2292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:35.877188 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:35.878834 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:35.878840 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:35.879012 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:35.879194 (+ 182us) client_negotiation.cc:769] Sending connection context
0807 07:01:35.879293 (+ 99us) client_negotiation.cc:240] Negotiation successful
0807 07:01:35.879430 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"mutex_wait_us":35,"thread_start_us":376,"threads_started":1}
I20260807 07:01:35.879905 25349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:35.872784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:58349 (local address 127.21.83.3:39465)
0807 07:01:35.873064 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:35.873070 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:35.873468 (+ 398us) server_negotiation.cc:407] Connection header received
0807 07:01:35.873666 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:35.873671 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:35.873728 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:35.873799 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:01:35.874343 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:35.875196 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:35.877652 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:35.878853 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:35.878981 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:35.879194 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:35.879529 (+ 335us) server_negotiation.cc:299] Negotiation successful
0807 07:01:35.879685 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 07:01:35.880587 25390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:35.872634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.3:39465 (local address 127.21.83.2:58349)
0807 07:01:35.873369 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:35.873383 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:35.873533 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:35.873826 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:35.873830 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:35.873853 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:01:35.874166 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:35.874175 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:35.875346 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:35.875355 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:35.877477 (+ 2122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:35.877488 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:35.879113 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:35.879122 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:35.879342 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:35.879375 (+ 33us) client_negotiation.cc:769] Sending connection context
0807 07:01:35.880075 (+ 700us) client_negotiation.cc:240] Negotiation successful
0807 07:01:35.880220 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"mutex_wait_us":52,"thread_start_us":375,"threads_started":1}
I20260807 07:01:35.880679 24807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64e108423d4f4cd5addf58e34d573c19" candidate_uuid: "633cabbcad0347b698839df57695c6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62aacfc55612476eac88f5a78966f57e" is_pre_election: true
I20260807 07:01:35.880995 24807 raft_consensus.cc:2454] T 64e108423d4f4cd5addf58e34d573c19 P 62aacfc55612476eac88f5a78966f57e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 633cabbcad0347b698839df57695c6cf in term 0.
I20260807 07:01:35.881476 24919 leader_election.cc:302] T 64e108423d4f4cd5addf58e34d573c19 P 633cabbcad0347b698839df57695c6cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62aacfc55612476eac88f5a78966f57e, 633cabbcad0347b698839df57695c6cf; no voters:
I20260807 07:01:35.881515 25247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64e108423d4f4cd5addf58e34d573c19" candidate_uuid: "633cabbcad0347b698839df57695c6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" is_pre_election: true
I20260807 07:01:35.881888 25363 raft_consensus.cc:2781] T 64e108423d4f4cd5addf58e34d573c19 P 633cabbcad0347b698839df57695c6cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 07:01:35.881997 25363 raft_consensus.cc:482] T 64e108423d4f4cd5addf58e34d573c19 P 633cabbcad0347b698839df57695c6cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:01:35.882064 25363 raft_consensus.cc:3037] T 64e108423d4f4cd5addf58e34d573c19 P 633cabbcad0347b698839df57695c6cf [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:01:35.882031 25247 raft_consensus.cc:2454] T 64e108423d4f4cd5addf58e34d573c19 P 8b9c6ea9dc714845ad211e5fd0e146cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 633cabbcad0347b698839df57695c6cf in term 0.
I20260807 07:01:35.884384 25363 raft_consensus.cc:504] T 64e108423d4f4cd5addf58e34d573c19 P 633cabbcad0347b698839df57695c6cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } }
I20260807 07:01:35.885052 25363 leader_election.cc:288] T 64e108423d4f4cd5addf58e34d573c19 P 633cabbcad0347b698839df57695c6cf [CANDIDATE]: Term 1 election: Requested vote from peers 62aacfc55612476eac88f5a78966f57e (127.21.83.1:33999), 8b9c6ea9dc714845ad211e5fd0e146cb (127.21.83.3:39465)
I20260807 07:01:35.885551 24807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64e108423d4f4cd5addf58e34d573c19" candidate_uuid: "633cabbcad0347b698839df57695c6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62aacfc55612476eac88f5a78966f57e"
I20260807 07:01:35.885604 25247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64e108423d4f4cd5addf58e34d573c19" candidate_uuid: "633cabbcad0347b698839df57695c6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb"
I20260807 07:01:35.885776 24807 raft_consensus.cc:3037] T 64e108423d4f4cd5addf58e34d573c19 P 62aacfc55612476eac88f5a78966f57e [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:01:35.885809 25247 raft_consensus.cc:3037] T 64e108423d4f4cd5addf58e34d573c19 P 8b9c6ea9dc714845ad211e5fd0e146cb [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:01:35.888128 24807 raft_consensus.cc:2454] T 64e108423d4f4cd5addf58e34d573c19 P 62aacfc55612476eac88f5a78966f57e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 633cabbcad0347b698839df57695c6cf in term 1.
I20260807 07:01:35.888182 25247 raft_consensus.cc:2454] T 64e108423d4f4cd5addf58e34d573c19 P 8b9c6ea9dc714845ad211e5fd0e146cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 633cabbcad0347b698839df57695c6cf in term 1.
I20260807 07:01:35.888600 24919 leader_election.cc:302] T 64e108423d4f4cd5addf58e34d573c19 P 633cabbcad0347b698839df57695c6cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 633cabbcad0347b698839df57695c6cf, 8b9c6ea9dc714845ad211e5fd0e146cb; no voters:
I20260807 07:01:35.889132 25363 raft_consensus.cc:2781] T 64e108423d4f4cd5addf58e34d573c19 P 633cabbcad0347b698839df57695c6cf [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:01:35.889843 25363 raft_consensus.cc:686] T 64e108423d4f4cd5addf58e34d573c19 P 633cabbcad0347b698839df57695c6cf [term 1 LEADER]: Becoming Leader. State: Replica: 633cabbcad0347b698839df57695c6cf, State: Running, Role: LEADER
I20260807 07:01:35.890441 25363 consensus_queue.cc:237] T 64e108423d4f4cd5addf58e34d573c19 P 633cabbcad0347b698839df57695c6cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } }
I20260807 07:01:35.894474 25391 tablet_replica.cc:439] T 64e108423d4f4cd5addf58e34d573c19 P 633cabbcad0347b698839df57695c6cf: TxnStatusTablet state changed. Reason: New leader 633cabbcad0347b698839df57695c6cf. Latest consensus state: current_term: 1 leader_uuid: "633cabbcad0347b698839df57695c6cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } } }
I20260807 07:01:35.894850 25391 tablet_replica.cc:442] T 64e108423d4f4cd5addf58e34d573c19 P 633cabbcad0347b698839df57695c6cf: This TxnStatusTablet replica's current role is: LEADER
I20260807 07:01:35.895629 25393 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 07:01:35.895884 25393 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260807 07:01:35.896520 24559 catalog_manager.cc:5495] T 64e108423d4f4cd5addf58e34d573c19 P 633cabbcad0347b698839df57695c6cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 633cabbcad0347b698839df57695c6cf (127.21.83.2). New cstate: current_term: 1 leader_uuid: "633cabbcad0347b698839df57695c6cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } health_report { overall_health: UNKNOWN } } }
I20260807 07:01:35.926069 25345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:35.919335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42652 (local address 127.21.83.2:43979)
0807 07:01:35.919619 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:35.919625 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:35.919650 (+ 25us) server_negotiation.cc:407] Connection header received
0807 07:01:35.919991 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:35.919995 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:35.920151 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:35.920280 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:01:35.920770 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:35.921928 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:35.923269 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:35.923709 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:35.923828 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:35.924310 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:01:35.924984 (+ 674us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:01:35.925131 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:35.925241 (+ 110us) server_negotiation.cc:299] Negotiation successful
0807 07:01:35.925363 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260807 07:01:35.926107 25379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:35.919284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.2:43979 (local address 127.0.0.1:42652)
0807 07:01:35.919577 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:35.919601 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:35.919766 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:35.920324 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:35.920329 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:35.920341 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 07:01:35.920535 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:35.920544 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:35.922083 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:35.922089 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:35.923121 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:35.923135 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:35.923870 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:35.923877 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:35.924045 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:35.924109 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 07:01:35.925102 (+ 993us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 07:01:35.925128 (+ 26us) client_negotiation.cc:769] Sending connection context
0807 07:01:35.925209 (+ 81us) client_negotiation.cc:240] Negotiation successful
0807 07:01:35.925382 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260807 07:01:35.934734 25247 raft_consensus.cc:1261] T 64e108423d4f4cd5addf58e34d573c19 P 8b9c6ea9dc714845ad211e5fd0e146cb [term 1 FOLLOWER]: Refusing update from remote peer 633cabbcad0347b698839df57695c6cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:01:35.934922 24807 raft_consensus.cc:1261] T 64e108423d4f4cd5addf58e34d573c19 P 62aacfc55612476eac88f5a78966f57e [term 1 FOLLOWER]: Refusing update from remote peer 633cabbcad0347b698839df57695c6cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:01:35.935850 25391 consensus_queue.cc:1035] T 64e108423d4f4cd5addf58e34d573c19 P 633cabbcad0347b698839df57695c6cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:01:35.936344 25363 consensus_queue.cc:1035] T 64e108423d4f4cd5addf58e34d573c19 P 633cabbcad0347b698839df57695c6cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:01:35.961334 25361 tablet_replica.cc:439] T 64e108423d4f4cd5addf58e34d573c19 P 62aacfc55612476eac88f5a78966f57e: TxnStatusTablet state changed. Reason: New leader 633cabbcad0347b698839df57695c6cf. Latest consensus state: current_term: 1 leader_uuid: "633cabbcad0347b698839df57695c6cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } } }
I20260807 07:01:35.961669 25361 tablet_replica.cc:442] T 64e108423d4f4cd5addf58e34d573c19 P 62aacfc55612476eac88f5a78966f57e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 07:01:35.963094 25365 tablet_replica.cc:439] T 64e108423d4f4cd5addf58e34d573c19 P 8b9c6ea9dc714845ad211e5fd0e146cb: TxnStatusTablet state changed. Reason: New leader 633cabbcad0347b698839df57695c6cf. Latest consensus state: current_term: 1 leader_uuid: "633cabbcad0347b698839df57695c6cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } } }
I20260807 07:01:35.963394 25365 tablet_replica.cc:442] T 64e108423d4f4cd5addf58e34d573c19 P 8b9c6ea9dc714845ad211e5fd0e146cb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 07:01:35.967247 25363 tablet_replica.cc:439] T 64e108423d4f4cd5addf58e34d573c19 P 633cabbcad0347b698839df57695c6cf: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "633cabbcad0347b698839df57695c6cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } } }
I20260807 07:01:35.967609 25363 tablet_replica.cc:442] T 64e108423d4f4cd5addf58e34d573c19 P 633cabbcad0347b698839df57695c6cf: This TxnStatusTablet replica's current role is: LEADER
I20260807 07:01:35.967857 25363 tablet_replica.cc:439] T 64e108423d4f4cd5addf58e34d573c19 P 633cabbcad0347b698839df57695c6cf: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "633cabbcad0347b698839df57695c6cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } } }
I20260807 07:01:35.968068 25363 tablet_replica.cc:442] T 64e108423d4f4cd5addf58e34d573c19 P 633cabbcad0347b698839df57695c6cf: This TxnStatusTablet replica's current role is: LEADER
I20260807 07:01:35.971597 25402 mvcc.cc:205] Tried to move back new op lower bound from 7315808648938369024 to 7315808648773283840. Current Snapshot: MvccSnapshot[applied={T|T < 7315808648938369024}]
I20260807 07:01:35.987610 25361 tablet_replica.cc:439] T 64e108423d4f4cd5addf58e34d573c19 P 62aacfc55612476eac88f5a78966f57e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "633cabbcad0347b698839df57695c6cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } } }
I20260807 07:01:35.987912 25361 tablet_replica.cc:442] T 64e108423d4f4cd5addf58e34d573c19 P 62aacfc55612476eac88f5a78966f57e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 07:01:35.989923 25365 tablet_replica.cc:439] T 64e108423d4f4cd5addf58e34d573c19 P 8b9c6ea9dc714845ad211e5fd0e146cb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "633cabbcad0347b698839df57695c6cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } } }
I20260807 07:01:35.990227 25365 tablet_replica.cc:442] T 64e108423d4f4cd5addf58e34d573c19 P 8b9c6ea9dc714845ad211e5fd0e146cb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 07:01:35.995980 25349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:35.987840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47550 (local address 127.21.83.3:39465)
0807 07:01:35.988001 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:35.988007 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:35.988039 (+ 32us) server_negotiation.cc:407] Connection header received
0807 07:01:35.988264 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:35.988270 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:35.988391 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:35.988549 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:01:35.990693 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:35.991891 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:35.993792 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:35.994200 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:35.994313 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:35.994803 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:01:35.995338 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:01:35.995469 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:35.995597 (+ 128us) server_negotiation.cc:299] Negotiation successful
0807 07:01:35.995687 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 07:01:36.013868 25418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:36.007297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.2:43979 (local address 127.21.83.3:48667)
0807 07:01:36.007998 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:36.008015 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:36.008153 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:36.008514 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:36.008517 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:36.008526 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 07:01:36.008951 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:36.008963 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:36.010390 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:36.010393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:36.011447 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:36.011459 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:36.012474 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:36.012479 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:36.012606 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:36.012677 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 07:01:36.013236 (+ 559us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 07:01:36.013249 (+ 13us) client_negotiation.cc:769] Sending connection context
0807 07:01:36.013329 (+ 80us) client_negotiation.cc:240] Negotiation successful
0807 07:01:36.013458 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":355,"threads_started":1}
I20260807 07:01:36.013860 25345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:36.007404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:48667 (local address 127.21.83.2:43979)
0807 07:01:36.007605 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:36.007612 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:36.008098 (+ 486us) server_negotiation.cc:407] Connection header received
0807 07:01:36.008282 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:36.008287 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:36.008404 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:36.008511 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:01:36.009113 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:36.010279 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:36.011583 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:36.012283 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:36.012416 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:36.012935 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:01:36.013092 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:01:36.013209 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:36.013371 (+ 162us) server_negotiation.cc:299] Negotiation successful
0807 07:01:36.013465 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260807 07:01:36.025270 24807 raft_consensus.cc:1261] T c071b3f1a36d46f4b02024cba222f54b P 62aacfc55612476eac88f5a78966f57e [term 1 FOLLOWER]: Refusing update from remote peer 8b9c6ea9dc714845ad211e5fd0e146cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:01:36.025403 25031 raft_consensus.cc:1261] T c071b3f1a36d46f4b02024cba222f54b P 633cabbcad0347b698839df57695c6cf [term 1 FOLLOWER]: Refusing update from remote peer 8b9c6ea9dc714845ad211e5fd0e146cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:01:36.026223 25368 consensus_queue.cc:1035] T c071b3f1a36d46f4b02024cba222f54b P 8b9c6ea9dc714845ad211e5fd0e146cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:01:36.026573 25365 consensus_queue.cc:1035] T c071b3f1a36d46f4b02024cba222f54b P 8b9c6ea9dc714845ad211e5fd0e146cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:01:36.069743 25349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:36.062486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:48419 (local address 127.21.83.3:39465)
0807 07:01:36.062669 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:36.062674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:36.063387 (+ 713us) server_negotiation.cc:407] Connection header received
0807 07:01:36.063594 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:36.063600 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:36.063688 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:36.063778 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:01:36.064564 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:36.065448 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:36.067626 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:36.068087 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:36.068198 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:36.068832 (+ 634us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:01:36.069032 (+ 200us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:01:36.069100 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:36.069359 (+ 259us) server_negotiation.cc:299] Negotiation successful
0807 07:01:36.069455 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 07:01:36.070824 25425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:36.062533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.3:39465 (local address 127.21.83.2:48419)
0807 07:01:36.063279 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:36.063299 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:36.063438 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:36.063945 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:36.063950 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:36.063964 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 07:01:36.064393 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:36.064406 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:36.065578 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:36.065584 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:36.067458 (+ 1874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:36.067478 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:36.068290 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:36.068297 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:36.068496 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:36.068601 (+ 105us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 07:01:36.069206 (+ 605us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 07:01:36.069218 (+ 12us) client_negotiation.cc:769] Sending connection context
0807 07:01:36.070186 (+ 968us) client_negotiation.cc:240] Negotiation successful
0807 07:01:36.070348 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":505,"threads_started":1}
I20260807 07:01:36.112684 24560 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:01:36.113040 24560 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:01:36.117408 24560 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bd7c08a8e2434a539ae91b39b7c750dc: Sending DeleteTablet for 3 replicas of tablet c071b3f1a36d46f4b02024cba222f54b
I20260807 07:01:36.118778 24766 tablet_service.cc:1514] Processing DeleteTablet for tablet c071b3f1a36d46f4b02024cba222f54b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:01:36 UTC) from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:35030
I20260807 07:01:36.118952 25000 tablet_service.cc:1514] Processing DeleteTablet for tablet c071b3f1a36d46f4b02024cba222f54b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:01:36 UTC) from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:42646
I20260807 07:01:36.120694 25210 tablet_service.cc:1514] Processing DeleteTablet for tablet c071b3f1a36d46f4b02024cba222f54b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:01:36 UTC) from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:47534
I20260807 07:01:36.130157 24559 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37284:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 07:01:36.130931 24559 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 07:01:36.131803 25431 tablet_replica.cc:330] T c071b3f1a36d46f4b02024cba222f54b P 62aacfc55612476eac88f5a78966f57e: stopping tablet replica
I20260807 07:01:36.141050 24996 tablet_service.cc:1467] Processing CreateTablet for tablet 11e8ee711f2c4c3d91874f4ba8955ab1 (DEFAULT_TABLE table=test-table [id=9e70fd26e35b4bd6b18e916133a35109]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:01:36.141697 24996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11e8ee711f2c4c3d91874f4ba8955ab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:01:36.143429 24766 tablet_service.cc:1467] Processing CreateTablet for tablet 11e8ee711f2c4c3d91874f4ba8955ab1 (DEFAULT_TABLE table=test-table [id=9e70fd26e35b4bd6b18e916133a35109]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:01:36.144030 24766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11e8ee711f2c4c3d91874f4ba8955ab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:01:36.148018 25433 tablet_replica.cc:330] T c071b3f1a36d46f4b02024cba222f54b P 8b9c6ea9dc714845ad211e5fd0e146cb: stopping tablet replica
I20260807 07:01:36.148489 25433 raft_consensus.cc:2229] T c071b3f1a36d46f4b02024cba222f54b P 8b9c6ea9dc714845ad211e5fd0e146cb [term 1 LEADER]: Raft consensus shutting down.
I20260807 07:01:36.149477 25432 tablet_replica.cc:330] T c071b3f1a36d46f4b02024cba222f54b P 633cabbcad0347b698839df57695c6cf: stopping tablet replica
I20260807 07:01:36.149413 25214 tablet_service.cc:1467] Processing CreateTablet for tablet 11e8ee711f2c4c3d91874f4ba8955ab1 (DEFAULT_TABLE table=test-table [id=9e70fd26e35b4bd6b18e916133a35109]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:01:36.149988 25214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11e8ee711f2c4c3d91874f4ba8955ab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:01:36.149915 25432 raft_consensus.cc:2229] T c071b3f1a36d46f4b02024cba222f54b P 633cabbcad0347b698839df57695c6cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:01:36.150300 25432 raft_consensus.cc:2258] T c071b3f1a36d46f4b02024cba222f54b P 633cabbcad0347b698839df57695c6cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:01:36.150007 25431 raft_consensus.cc:2229] T c071b3f1a36d46f4b02024cba222f54b P 62aacfc55612476eac88f5a78966f57e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:01:36.151437 25433 raft_consensus.cc:2258] T c071b3f1a36d46f4b02024cba222f54b P 8b9c6ea9dc714845ad211e5fd0e146cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:01:36.153770 25433 ts_tablet_manager.cc:1905] T c071b3f1a36d46f4b02024cba222f54b P 8b9c6ea9dc714845ad211e5fd0e146cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:01:36.154040 25431 raft_consensus.cc:2258] T c071b3f1a36d46f4b02024cba222f54b P 62aacfc55612476eac88f5a78966f57e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:01:36.156716 25358 tablet_bootstrap.cc:492] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 8b9c6ea9dc714845ad211e5fd0e146cb: Bootstrap starting.
I20260807 07:01:36.157797 25432 ts_tablet_manager.cc:1905] T c071b3f1a36d46f4b02024cba222f54b P 633cabbcad0347b698839df57695c6cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:01:36.158066 25431 ts_tablet_manager.cc:1905] T c071b3f1a36d46f4b02024cba222f54b P 62aacfc55612476eac88f5a78966f57e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:01:36.158418 25358 tablet_bootstrap.cc:654] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 8b9c6ea9dc714845ad211e5fd0e146cb: Neither blocks nor log segments found. Creating new log.
I20260807 07:01:36.162952 25352 tablet_bootstrap.cc:492] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 633cabbcad0347b698839df57695c6cf: Bootstrap starting.
I20260807 07:01:36.164917 25352 tablet_bootstrap.cc:654] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 633cabbcad0347b698839df57695c6cf: Neither blocks nor log segments found. Creating new log.
I20260807 07:01:36.165720 25350 tablet_bootstrap.cc:492] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 62aacfc55612476eac88f5a78966f57e: Bootstrap starting.
I20260807 07:01:36.167310 25433 ts_tablet_manager.cc:1918] T c071b3f1a36d46f4b02024cba222f54b P 8b9c6ea9dc714845ad211e5fd0e146cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 07:01:36.167491 25433 log.cc:1196] T c071b3f1a36d46f4b02024cba222f54b P 8b9c6ea9dc714845ad211e5fd0e146cb: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-2/wal/wals/c071b3f1a36d46f4b02024cba222f54b
I20260807 07:01:36.167519 25350 tablet_bootstrap.cc:654] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 62aacfc55612476eac88f5a78966f57e: Neither blocks nor log segments found. Creating new log.
I20260807 07:01:36.168022 25433 ts_tablet_manager.cc:1939] T c071b3f1a36d46f4b02024cba222f54b P 8b9c6ea9dc714845ad211e5fd0e146cb: Deleting consensus metadata
I20260807 07:01:36.170042 24538 catalog_manager.cc:4841] TS 8b9c6ea9dc714845ad211e5fd0e146cb (127.21.83.3:39465): tablet c071b3f1a36d46f4b02024cba222f54b (table test-table [id=bb8976c963184c3b8e767db911321112]) successfully deleted
I20260807 07:01:36.175506 25432 ts_tablet_manager.cc:1918] T c071b3f1a36d46f4b02024cba222f54b P 633cabbcad0347b698839df57695c6cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 07:01:36.175781 25432 log.cc:1196] T c071b3f1a36d46f4b02024cba222f54b P 633cabbcad0347b698839df57695c6cf: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-1/wal/wals/c071b3f1a36d46f4b02024cba222f54b
I20260807 07:01:36.176296 25432 ts_tablet_manager.cc:1939] T c071b3f1a36d46f4b02024cba222f54b P 633cabbcad0347b698839df57695c6cf: Deleting consensus metadata
I20260807 07:01:36.178313 24543 catalog_manager.cc:4841] TS 633cabbcad0347b698839df57695c6cf (127.21.83.2:43979): tablet c071b3f1a36d46f4b02024cba222f54b (table test-table [id=bb8976c963184c3b8e767db911321112]) successfully deleted
I20260807 07:01:36.185693 25350 tablet_bootstrap.cc:492] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 62aacfc55612476eac88f5a78966f57e: No bootstrap required, opened a new log
I20260807 07:01:36.185902 25350 ts_tablet_manager.cc:1397] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 62aacfc55612476eac88f5a78966f57e: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.006s
I20260807 07:01:36.186817 25350 raft_consensus.cc:348] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 62aacfc55612476eac88f5a78966f57e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } }
I20260807 07:01:36.187079 25350 raft_consensus.cc:374] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 62aacfc55612476eac88f5a78966f57e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:01:36.187145 25350 raft_consensus.cc:729] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 62aacfc55612476eac88f5a78966f57e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62aacfc55612476eac88f5a78966f57e, State: Initialized, Role: FOLLOWER
I20260807 07:01:36.187418 25350 consensus_queue.cc:260] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 62aacfc55612476eac88f5a78966f57e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } }
I20260807 07:01:36.188081 25350 ts_tablet_manager.cc:1428] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 62aacfc55612476eac88f5a78966f57e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260807 07:01:36.188445 25352 tablet_bootstrap.cc:492] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 633cabbcad0347b698839df57695c6cf: No bootstrap required, opened a new log
I20260807 07:01:36.188632 25352 ts_tablet_manager.cc:1397] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 633cabbcad0347b698839df57695c6cf: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.008s
I20260807 07:01:36.189455 25431 ts_tablet_manager.cc:1918] T c071b3f1a36d46f4b02024cba222f54b P 62aacfc55612476eac88f5a78966f57e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 07:01:36.189637 25431 log.cc:1196] T c071b3f1a36d46f4b02024cba222f54b P 62aacfc55612476eac88f5a78966f57e: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-0/wal/wals/c071b3f1a36d46f4b02024cba222f54b
I20260807 07:01:36.189502 25352 raft_consensus.cc:348] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 633cabbcad0347b698839df57695c6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } }
I20260807 07:01:36.189769 25352 raft_consensus.cc:374] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 633cabbcad0347b698839df57695c6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:01:36.189846 25352 raft_consensus.cc:729] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 633cabbcad0347b698839df57695c6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 633cabbcad0347b698839df57695c6cf, State: Initialized, Role: FOLLOWER
I20260807 07:01:36.190150 25431 ts_tablet_manager.cc:1939] T c071b3f1a36d46f4b02024cba222f54b P 62aacfc55612476eac88f5a78966f57e: Deleting consensus metadata
I20260807 07:01:36.190776 25358 tablet_bootstrap.cc:492] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 8b9c6ea9dc714845ad211e5fd0e146cb: No bootstrap required, opened a new log
I20260807 07:01:36.190972 25358 ts_tablet_manager.cc:1397] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 8b9c6ea9dc714845ad211e5fd0e146cb: Time spent bootstrapping tablet: real 0.034s user 0.007s sys 0.005s
I20260807 07:01:36.191745 25358 raft_consensus.cc:348] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 8b9c6ea9dc714845ad211e5fd0e146cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } }
I20260807 07:01:36.191988 25358 raft_consensus.cc:374] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 8b9c6ea9dc714845ad211e5fd0e146cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:01:36.192070 25358 raft_consensus.cc:729] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 8b9c6ea9dc714845ad211e5fd0e146cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b9c6ea9dc714845ad211e5fd0e146cb, State: Initialized, Role: FOLLOWER
I20260807 07:01:36.192128 24538 catalog_manager.cc:4841] TS 62aacfc55612476eac88f5a78966f57e (127.21.83.1:33999): tablet c071b3f1a36d46f4b02024cba222f54b (table test-table [id=bb8976c963184c3b8e767db911321112]) successfully deleted
I20260807 07:01:36.192314 25358 consensus_queue.cc:260] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 8b9c6ea9dc714845ad211e5fd0e146cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } }
I20260807 07:01:36.192703 25352 consensus_queue.cc:260] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 633cabbcad0347b698839df57695c6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } }
I20260807 07:01:36.193521 25352 ts_tablet_manager.cc:1428] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 633cabbcad0347b698839df57695c6cf: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260807 07:01:36.197201 25358 ts_tablet_manager.cc:1428] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 8b9c6ea9dc714845ad211e5fd0e146cb: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20260807 07:01:36.380168 25365 raft_consensus.cc:482] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 8b9c6ea9dc714845ad211e5fd0e146cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:01:36.380402 25365 raft_consensus.cc:504] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 8b9c6ea9dc714845ad211e5fd0e146cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } }
I20260807 07:01:36.381258 25365 leader_election.cc:288] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 8b9c6ea9dc714845ad211e5fd0e146cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62aacfc55612476eac88f5a78966f57e (127.21.83.1:33999), 633cabbcad0347b698839df57695c6cf (127.21.83.2:43979)
I20260807 07:01:36.381846 24804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11e8ee711f2c4c3d91874f4ba8955ab1" candidate_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62aacfc55612476eac88f5a78966f57e" is_pre_election: true
I20260807 07:01:36.381944 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11e8ee711f2c4c3d91874f4ba8955ab1" candidate_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "633cabbcad0347b698839df57695c6cf" is_pre_election: true
I20260807 07:01:36.382108 24804 raft_consensus.cc:2454] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 62aacfc55612476eac88f5a78966f57e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b9c6ea9dc714845ad211e5fd0e146cb in term 0.
I20260807 07:01:36.382229 25029 raft_consensus.cc:2454] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 633cabbcad0347b698839df57695c6cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b9c6ea9dc714845ad211e5fd0e146cb in term 0.
I20260807 07:01:36.382649 25148 leader_election.cc:302] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 8b9c6ea9dc714845ad211e5fd0e146cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 633cabbcad0347b698839df57695c6cf, 8b9c6ea9dc714845ad211e5fd0e146cb; no voters:
I20260807 07:01:36.382925 25365 raft_consensus.cc:2781] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 8b9c6ea9dc714845ad211e5fd0e146cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 07:01:36.383085 25365 raft_consensus.cc:482] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 8b9c6ea9dc714845ad211e5fd0e146cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:01:36.383180 25365 raft_consensus.cc:3037] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 8b9c6ea9dc714845ad211e5fd0e146cb [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:01:36.385538 25365 raft_consensus.cc:504] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 8b9c6ea9dc714845ad211e5fd0e146cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } }
I20260807 07:01:36.386128 25365 leader_election.cc:288] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 8b9c6ea9dc714845ad211e5fd0e146cb [CANDIDATE]: Term 1 election: Requested vote from peers 62aacfc55612476eac88f5a78966f57e (127.21.83.1:33999), 633cabbcad0347b698839df57695c6cf (127.21.83.2:43979)
I20260807 07:01:36.386718 24804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11e8ee711f2c4c3d91874f4ba8955ab1" candidate_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62aacfc55612476eac88f5a78966f57e"
I20260807 07:01:36.386772 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11e8ee711f2c4c3d91874f4ba8955ab1" candidate_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "633cabbcad0347b698839df57695c6cf"
I20260807 07:01:36.386940 24804 raft_consensus.cc:3037] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 62aacfc55612476eac88f5a78966f57e [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:01:36.387019 25029 raft_consensus.cc:3037] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 633cabbcad0347b698839df57695c6cf [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:01:36.389194 24804 raft_consensus.cc:2454] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 62aacfc55612476eac88f5a78966f57e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b9c6ea9dc714845ad211e5fd0e146cb in term 1.
I20260807 07:01:36.389998 25029 raft_consensus.cc:2454] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 633cabbcad0347b698839df57695c6cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b9c6ea9dc714845ad211e5fd0e146cb in term 1.
I20260807 07:01:36.390492 25148 leader_election.cc:302] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 8b9c6ea9dc714845ad211e5fd0e146cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 633cabbcad0347b698839df57695c6cf, 8b9c6ea9dc714845ad211e5fd0e146cb; no voters:
I20260807 07:01:36.390821 25365 raft_consensus.cc:2781] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 8b9c6ea9dc714845ad211e5fd0e146cb [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:01:36.391038 25365 raft_consensus.cc:686] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 8b9c6ea9dc714845ad211e5fd0e146cb [term 1 LEADER]: Becoming Leader. State: Replica: 8b9c6ea9dc714845ad211e5fd0e146cb, State: Running, Role: LEADER
I20260807 07:01:36.391398 25365 consensus_queue.cc:237] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 8b9c6ea9dc714845ad211e5fd0e146cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } }
I20260807 07:01:36.394233 24559 catalog_manager.cc:5495] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 8b9c6ea9dc714845ad211e5fd0e146cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b9c6ea9dc714845ad211e5fd0e146cb (127.21.83.3). New cstate: current_term: 1 leader_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } health_report { overall_health: HEALTHY } } }
I20260807 07:01:36.467962 25029 raft_consensus.cc:1261] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 633cabbcad0347b698839df57695c6cf [term 1 FOLLOWER]: Refusing update from remote peer 8b9c6ea9dc714845ad211e5fd0e146cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:01:36.468556 25365 consensus_queue.cc:1035] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 8b9c6ea9dc714845ad211e5fd0e146cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:01:36.469075 24804 raft_consensus.cc:1261] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 62aacfc55612476eac88f5a78966f57e [term 1 FOLLOWER]: Refusing update from remote peer 8b9c6ea9dc714845ad211e5fd0e146cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:01:36.469607 25365 consensus_queue.cc:1035] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 8b9c6ea9dc714845ad211e5fd0e146cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:01:36.474527 25401 mvcc.cc:205] Tried to move back new op lower bound from 7315808651127783424 to 7315808650821947392. Current Snapshot: MvccSnapshot[applied={T|T < 7315808651127783424}]
I20260807 07:01:36.490051 25423 mvcc.cc:205] Tried to move back new op lower bound from 7315808651127783424 to 7315808650821947392. Current Snapshot: MvccSnapshot[applied={T|T < 7315808651127783424}]
W20260807 07:01:36.514249 25401 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260807 07:01:38.109074 24558 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:01:38.109294 24558 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:01:38.113359 24558 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bd7c08a8e2434a539ae91b39b7c750dc: Sending DeleteTablet for 3 replicas of tablet 11e8ee711f2c4c3d91874f4ba8955ab1
I20260807 07:01:38.114157 24766 tablet_service.cc:1514] Processing DeleteTablet for tablet 11e8ee711f2c4c3d91874f4ba8955ab1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:01:38 UTC) from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:35030
I20260807 07:01:38.114210 25214 tablet_service.cc:1514] Processing DeleteTablet for tablet 11e8ee711f2c4c3d91874f4ba8955ab1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:01:38 UTC) from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:47534
I20260807 07:01:38.114245 24996 tablet_service.cc:1514] Processing DeleteTablet for tablet 11e8ee711f2c4c3d91874f4ba8955ab1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:01:38 UTC) from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:42646
I20260807 07:01:38.126734 25486 tablet_replica.cc:330] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 633cabbcad0347b698839df57695c6cf: stopping tablet replica
I20260807 07:01:38.126886 25484 tablet_replica.cc:330] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 62aacfc55612476eac88f5a78966f57e: stopping tablet replica
I20260807 07:01:38.127177 25484 raft_consensus.cc:2229] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 62aacfc55612476eac88f5a78966f57e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:01:38.127187 25486 raft_consensus.cc:2229] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 633cabbcad0347b698839df57695c6cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:01:38.127470 25486 raft_consensus.cc:2258] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 633cabbcad0347b698839df57695c6cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:01:38.127506 25484 raft_consensus.cc:2258] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 62aacfc55612476eac88f5a78966f57e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:01:38.128625 25484 ts_tablet_manager.cc:1905] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 62aacfc55612476eac88f5a78966f57e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:01:38.128727 25486 ts_tablet_manager.cc:1905] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 633cabbcad0347b698839df57695c6cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:01:38.135172 25484 ts_tablet_manager.cc:1918] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 62aacfc55612476eac88f5a78966f57e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 07:01:38.135443 25484 log.cc:1196] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 62aacfc55612476eac88f5a78966f57e: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-0/wal/wals/11e8ee711f2c4c3d91874f4ba8955ab1
I20260807 07:01:38.135953 25484 ts_tablet_manager.cc:1939] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 62aacfc55612476eac88f5a78966f57e: Deleting consensus metadata
I20260807 07:01:38.136207 25486 ts_tablet_manager.cc:1918] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 633cabbcad0347b698839df57695c6cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 07:01:38.136782 25485 tablet_replica.cc:330] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 8b9c6ea9dc714845ad211e5fd0e146cb: stopping tablet replica
I20260807 07:01:38.137013 25485 raft_consensus.cc:2229] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 8b9c6ea9dc714845ad211e5fd0e146cb [term 1 LEADER]: Raft consensus shutting down.
I20260807 07:01:38.137259 24538 catalog_manager.cc:4841] TS 62aacfc55612476eac88f5a78966f57e (127.21.83.1:33999): tablet 11e8ee711f2c4c3d91874f4ba8955ab1 (table test-table [id=9e70fd26e35b4bd6b18e916133a35109]) successfully deleted
I20260807 07:01:38.137473 25485 raft_consensus.cc:2258] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 8b9c6ea9dc714845ad211e5fd0e146cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:01:38.138307 25485 ts_tablet_manager.cc:1905] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 8b9c6ea9dc714845ad211e5fd0e146cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:01:38.142788 25485 ts_tablet_manager.cc:1918] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 8b9c6ea9dc714845ad211e5fd0e146cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 07:01:38.142923 25485 log.cc:1196] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 8b9c6ea9dc714845ad211e5fd0e146cb: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-2/wal/wals/11e8ee711f2c4c3d91874f4ba8955ab1
I20260807 07:01:38.142987 25486 log.cc:1196] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 633cabbcad0347b698839df57695c6cf: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-1/wal/wals/11e8ee711f2c4c3d91874f4ba8955ab1
I20260807 07:01:38.143401 25485 ts_tablet_manager.cc:1939] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 8b9c6ea9dc714845ad211e5fd0e146cb: Deleting consensus metadata
I20260807 07:01:38.143483 25486 ts_tablet_manager.cc:1939] T 11e8ee711f2c4c3d91874f4ba8955ab1 P 633cabbcad0347b698839df57695c6cf: Deleting consensus metadata
I20260807 07:01:38.144685 24543 catalog_manager.cc:4841] TS 633cabbcad0347b698839df57695c6cf (127.21.83.2:43979): tablet 11e8ee711f2c4c3d91874f4ba8955ab1 (table test-table [id=9e70fd26e35b4bd6b18e916133a35109]) successfully deleted
I20260807 07:01:38.144735 24538 catalog_manager.cc:4841] TS 8b9c6ea9dc714845ad211e5fd0e146cb (127.21.83.3:39465): tablet 11e8ee711f2c4c3d91874f4ba8955ab1 (table test-table [id=9e70fd26e35b4bd6b18e916133a35109]) successfully deleted
Aug 07 07:01:38 dist-test-slave-ncp9 krb5kdc[24491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086098, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 07:01:38 dist-test-slave-ncp9 krb5kdc[24491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086098, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:01:38.178428 25500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:38.163434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37306 (local address 127.21.83.62:34197)
0807 07:01:38.164031 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:38.164036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:38.164100 (+ 64us) server_negotiation.cc:407] Connection header received
0807 07:01:38.164160 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:38.164165 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:38.164229 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:38.164320 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:38.165764 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:38.166584 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:38.167941 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:38.168385 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:38.168497 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:38.172611 (+ 4114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:38.172618 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:38.172622 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:38.172670 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:38.175771 (+ 3101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:38.176595 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:38.176600 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:38.176602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:38.176755 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:38.177132 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:38.177136 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:38.177138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:38.177482 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:38.177697 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:38.177941 (+ 244us) server_negotiation.cc:299] Negotiation successful
0807 07:01:38.178134 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":315,"threads_started":1}
I20260807 07:01:38.181973 24558 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37306:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 07:01:38.182628 24558 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 07:01:38.189339 24766 tablet_service.cc:1467] Processing CreateTablet for tablet dc799afaa39e49c584562742b9477436 (DEFAULT_TABLE table=test-table [id=4b7211acc4204ea2b3d8ec391c4a5fd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:01:38.189561 24996 tablet_service.cc:1467] Processing CreateTablet for tablet dc799afaa39e49c584562742b9477436 (DEFAULT_TABLE table=test-table [id=4b7211acc4204ea2b3d8ec391c4a5fd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:01:38.189862 24766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc799afaa39e49c584562742b9477436. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:01:38.189996 24996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc799afaa39e49c584562742b9477436. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:01:38.195082 25214 tablet_service.cc:1467] Processing CreateTablet for tablet dc799afaa39e49c584562742b9477436 (DEFAULT_TABLE table=test-table [id=4b7211acc4204ea2b3d8ec391c4a5fd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:01:38.195305 25501 tablet_bootstrap.cc:492] T dc799afaa39e49c584562742b9477436 P 633cabbcad0347b698839df57695c6cf: Bootstrap starting.
I20260807 07:01:38.195550 25214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc799afaa39e49c584562742b9477436. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:01:38.195578 25502 tablet_bootstrap.cc:492] T dc799afaa39e49c584562742b9477436 P 62aacfc55612476eac88f5a78966f57e: Bootstrap starting.
I20260807 07:01:38.196689 25501 tablet_bootstrap.cc:654] T dc799afaa39e49c584562742b9477436 P 633cabbcad0347b698839df57695c6cf: Neither blocks nor log segments found. Creating new log.
I20260807 07:01:38.197238 25502 tablet_bootstrap.cc:654] T dc799afaa39e49c584562742b9477436 P 62aacfc55612476eac88f5a78966f57e: Neither blocks nor log segments found. Creating new log.
I20260807 07:01:38.199028 25501 tablet_bootstrap.cc:492] T dc799afaa39e49c584562742b9477436 P 633cabbcad0347b698839df57695c6cf: No bootstrap required, opened a new log
I20260807 07:01:38.199254 25501 ts_tablet_manager.cc:1397] T dc799afaa39e49c584562742b9477436 P 633cabbcad0347b698839df57695c6cf: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260807 07:01:38.200001 25501 raft_consensus.cc:348] T dc799afaa39e49c584562742b9477436 P 633cabbcad0347b698839df57695c6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } }
I20260807 07:01:38.200187 25501 raft_consensus.cc:374] T dc799afaa39e49c584562742b9477436 P 633cabbcad0347b698839df57695c6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:01:38.200294 25501 raft_consensus.cc:729] T dc799afaa39e49c584562742b9477436 P 633cabbcad0347b698839df57695c6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 633cabbcad0347b698839df57695c6cf, State: Initialized, Role: FOLLOWER
I20260807 07:01:38.200320 25502 tablet_bootstrap.cc:492] T dc799afaa39e49c584562742b9477436 P 62aacfc55612476eac88f5a78966f57e: No bootstrap required, opened a new log
I20260807 07:01:38.200519 25502 ts_tablet_manager.cc:1397] T dc799afaa39e49c584562742b9477436 P 62aacfc55612476eac88f5a78966f57e: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260807 07:01:38.200512 25501 consensus_queue.cc:260] T dc799afaa39e49c584562742b9477436 P 633cabbcad0347b698839df57695c6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } }
I20260807 07:01:38.201094 25501 ts_tablet_manager.cc:1428] T dc799afaa39e49c584562742b9477436 P 633cabbcad0347b698839df57695c6cf: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260807 07:01:38.201436 25502 raft_consensus.cc:348] T dc799afaa39e49c584562742b9477436 P 62aacfc55612476eac88f5a78966f57e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } }
I20260807 07:01:38.202368 25506 tablet_bootstrap.cc:492] T dc799afaa39e49c584562742b9477436 P 8b9c6ea9dc714845ad211e5fd0e146cb: Bootstrap starting.
I20260807 07:01:38.201675 25502 raft_consensus.cc:374] T dc799afaa39e49c584562742b9477436 P 62aacfc55612476eac88f5a78966f57e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:01:38.202921 25502 raft_consensus.cc:729] T dc799afaa39e49c584562742b9477436 P 62aacfc55612476eac88f5a78966f57e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62aacfc55612476eac88f5a78966f57e, State: Initialized, Role: FOLLOWER
I20260807 07:01:38.203186 25502 consensus_queue.cc:260] T dc799afaa39e49c584562742b9477436 P 62aacfc55612476eac88f5a78966f57e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } }
I20260807 07:01:38.204154 25502 ts_tablet_manager.cc:1428] T dc799afaa39e49c584562742b9477436 P 62aacfc55612476eac88f5a78966f57e: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260807 07:01:38.204628 25506 tablet_bootstrap.cc:654] T dc799afaa39e49c584562742b9477436 P 8b9c6ea9dc714845ad211e5fd0e146cb: Neither blocks nor log segments found. Creating new log.
I20260807 07:01:38.206853 25506 tablet_bootstrap.cc:492] T dc799afaa39e49c584562742b9477436 P 8b9c6ea9dc714845ad211e5fd0e146cb: No bootstrap required, opened a new log
I20260807 07:01:38.207001 25506 ts_tablet_manager.cc:1397] T dc799afaa39e49c584562742b9477436 P 8b9c6ea9dc714845ad211e5fd0e146cb: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260807 07:01:38.207762 25506 raft_consensus.cc:348] T dc799afaa39e49c584562742b9477436 P 8b9c6ea9dc714845ad211e5fd0e146cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } }
I20260807 07:01:38.207999 25506 raft_consensus.cc:374] T dc799afaa39e49c584562742b9477436 P 8b9c6ea9dc714845ad211e5fd0e146cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:01:38.208099 25506 raft_consensus.cc:729] T dc799afaa39e49c584562742b9477436 P 8b9c6ea9dc714845ad211e5fd0e146cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b9c6ea9dc714845ad211e5fd0e146cb, State: Initialized, Role: FOLLOWER
I20260807 07:01:38.208398 25506 consensus_queue.cc:260] T dc799afaa39e49c584562742b9477436 P 8b9c6ea9dc714845ad211e5fd0e146cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } }
I20260807 07:01:38.208946 25506 ts_tablet_manager.cc:1428] T dc799afaa39e49c584562742b9477436 P 8b9c6ea9dc714845ad211e5fd0e146cb: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260807 07:01:38.452361 25471 raft_consensus.cc:482] T dc799afaa39e49c584562742b9477436 P 8b9c6ea9dc714845ad211e5fd0e146cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:01:38.452600 25471 raft_consensus.cc:504] T dc799afaa39e49c584562742b9477436 P 8b9c6ea9dc714845ad211e5fd0e146cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } }
I20260807 07:01:38.453310 25471 leader_election.cc:288] T dc799afaa39e49c584562742b9477436 P 8b9c6ea9dc714845ad211e5fd0e146cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62aacfc55612476eac88f5a78966f57e (127.21.83.1:33999), 633cabbcad0347b698839df57695c6cf (127.21.83.2:43979)
I20260807 07:01:38.453986 24804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc799afaa39e49c584562742b9477436" candidate_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62aacfc55612476eac88f5a78966f57e" is_pre_election: true
I20260807 07:01:38.454025 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc799afaa39e49c584562742b9477436" candidate_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "633cabbcad0347b698839df57695c6cf" is_pre_election: true
I20260807 07:01:38.454224 24804 raft_consensus.cc:2454] T dc799afaa39e49c584562742b9477436 P 62aacfc55612476eac88f5a78966f57e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b9c6ea9dc714845ad211e5fd0e146cb in term 0.
I20260807 07:01:38.454267 25029 raft_consensus.cc:2454] T dc799afaa39e49c584562742b9477436 P 633cabbcad0347b698839df57695c6cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b9c6ea9dc714845ad211e5fd0e146cb in term 0.
I20260807 07:01:38.454741 25140 leader_election.cc:302] T dc799afaa39e49c584562742b9477436 P 8b9c6ea9dc714845ad211e5fd0e146cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 62aacfc55612476eac88f5a78966f57e, 633cabbcad0347b698839df57695c6cf, 8b9c6ea9dc714845ad211e5fd0e146cb; no voters:
I20260807 07:01:38.455042 25471 raft_consensus.cc:2781] T dc799afaa39e49c584562742b9477436 P 8b9c6ea9dc714845ad211e5fd0e146cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 07:01:38.455150 25471 raft_consensus.cc:482] T dc799afaa39e49c584562742b9477436 P 8b9c6ea9dc714845ad211e5fd0e146cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:01:38.455216 25471 raft_consensus.cc:3037] T dc799afaa39e49c584562742b9477436 P 8b9c6ea9dc714845ad211e5fd0e146cb [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:01:38.457441 25471 raft_consensus.cc:504] T dc799afaa39e49c584562742b9477436 P 8b9c6ea9dc714845ad211e5fd0e146cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } }
I20260807 07:01:38.458025 25471 leader_election.cc:288] T dc799afaa39e49c584562742b9477436 P 8b9c6ea9dc714845ad211e5fd0e146cb [CANDIDATE]: Term 1 election: Requested vote from peers 62aacfc55612476eac88f5a78966f57e (127.21.83.1:33999), 633cabbcad0347b698839df57695c6cf (127.21.83.2:43979)
I20260807 07:01:38.458575 24804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc799afaa39e49c584562742b9477436" candidate_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62aacfc55612476eac88f5a78966f57e"
I20260807 07:01:38.458611 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc799afaa39e49c584562742b9477436" candidate_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "633cabbcad0347b698839df57695c6cf"
I20260807 07:01:38.458812 24804 raft_consensus.cc:3037] T dc799afaa39e49c584562742b9477436 P 62aacfc55612476eac88f5a78966f57e [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:01:38.458835 25029 raft_consensus.cc:3037] T dc799afaa39e49c584562742b9477436 P 633cabbcad0347b698839df57695c6cf [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:01:38.461052 24804 raft_consensus.cc:2454] T dc799afaa39e49c584562742b9477436 P 62aacfc55612476eac88f5a78966f57e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b9c6ea9dc714845ad211e5fd0e146cb in term 1.
I20260807 07:01:38.461067 25029 raft_consensus.cc:2454] T dc799afaa39e49c584562742b9477436 P 633cabbcad0347b698839df57695c6cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b9c6ea9dc714845ad211e5fd0e146cb in term 1.
I20260807 07:01:38.461529 25148 leader_election.cc:302] T dc799afaa39e49c584562742b9477436 P 8b9c6ea9dc714845ad211e5fd0e146cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 633cabbcad0347b698839df57695c6cf, 8b9c6ea9dc714845ad211e5fd0e146cb; no voters:
I20260807 07:01:38.461931 25471 raft_consensus.cc:2781] T dc799afaa39e49c584562742b9477436 P 8b9c6ea9dc714845ad211e5fd0e146cb [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:01:38.462080 25471 raft_consensus.cc:686] T dc799afaa39e49c584562742b9477436 P 8b9c6ea9dc714845ad211e5fd0e146cb [term 1 LEADER]: Becoming Leader. State: Replica: 8b9c6ea9dc714845ad211e5fd0e146cb, State: Running, Role: LEADER
I20260807 07:01:38.462387 25471 consensus_queue.cc:237] T dc799afaa39e49c584562742b9477436 P 8b9c6ea9dc714845ad211e5fd0e146cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } }
I20260807 07:01:38.465096 24559 catalog_manager.cc:5495] T dc799afaa39e49c584562742b9477436 P 8b9c6ea9dc714845ad211e5fd0e146cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b9c6ea9dc714845ad211e5fd0e146cb (127.21.83.3). New cstate: current_term: 1 leader_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } health_report { overall_health: HEALTHY } } }
I20260807 07:01:38.502225 25514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:38.497298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47558 (local address 127.21.83.3:39465)
0807 07:01:38.497895 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:38.497899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:38.497916 (+ 17us) server_negotiation.cc:407] Connection header received
0807 07:01:38.497965 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:38.497969 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:38.498039 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:38.498120 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:01:38.498499 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:38.499256 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:38.500528 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:38.500972 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:38.501090 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:38.501517 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:01:38.501617 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:01:38.501690 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:38.501842 (+ 152us) server_negotiation.cc:299] Negotiation successful
0807 07:01:38.501918 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":321,"threads_started":1}
I20260807 07:01:38.510555 24804 raft_consensus.cc:1261] T dc799afaa39e49c584562742b9477436 P 62aacfc55612476eac88f5a78966f57e [term 1 FOLLOWER]: Refusing update from remote peer 8b9c6ea9dc714845ad211e5fd0e146cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:01:38.510618 25029 raft_consensus.cc:1261] T dc799afaa39e49c584562742b9477436 P 633cabbcad0347b698839df57695c6cf [term 1 FOLLOWER]: Refusing update from remote peer 8b9c6ea9dc714845ad211e5fd0e146cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:01:38.511137 25471 consensus_queue.cc:1035] T dc799afaa39e49c584562742b9477436 P 8b9c6ea9dc714845ad211e5fd0e146cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:01:38.511397 25465 consensus_queue.cc:1035] T dc799afaa39e49c584562742b9477436 P 8b9c6ea9dc714845ad211e5fd0e146cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:01:38.554340 24558 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:01:38.554581 24558 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:01:38.557215 24558 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bd7c08a8e2434a539ae91b39b7c750dc: Sending DeleteTablet for 3 replicas of tablet dc799afaa39e49c584562742b9477436
I20260807 07:01:38.557921 24996 tablet_service.cc:1514] Processing DeleteTablet for tablet dc799afaa39e49c584562742b9477436 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:01:38 UTC) from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:42646
I20260807 07:01:38.557950 24766 tablet_service.cc:1514] Processing DeleteTablet for tablet dc799afaa39e49c584562742b9477436 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:01:38 UTC) from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:35030
I20260807 07:01:38.558176 25214 tablet_service.cc:1514] Processing DeleteTablet for tablet dc799afaa39e49c584562742b9477436 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:01:38 UTC) from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:47534
I20260807 07:01:38.558431 25484 tablet_replica.cc:330] T dc799afaa39e49c584562742b9477436 P 62aacfc55612476eac88f5a78966f57e: stopping tablet replica
I20260807 07:01:38.558651 25484 raft_consensus.cc:2229] T dc799afaa39e49c584562742b9477436 P 62aacfc55612476eac88f5a78966f57e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:01:38.558821 25484 raft_consensus.cc:2258] T dc799afaa39e49c584562742b9477436 P 62aacfc55612476eac88f5a78966f57e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:01:38.558789 24558 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37306:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260807 07:01:38.559100 25485 tablet_replica.cc:330] T dc799afaa39e49c584562742b9477436 P 8b9c6ea9dc714845ad211e5fd0e146cb: stopping tablet replica
I20260807 07:01:38.559247 25485 raft_consensus.cc:2229] T dc799afaa39e49c584562742b9477436 P 8b9c6ea9dc714845ad211e5fd0e146cb [term 1 LEADER]: Raft consensus shutting down.
I20260807 07:01:38.559520 25485 raft_consensus.cc:2258] T dc799afaa39e49c584562742b9477436 P 8b9c6ea9dc714845ad211e5fd0e146cb [term 1 FOLLOWER]: Raft consensus is shut down!
W20260807 07:01:38.559520 24558 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 07:01:38.559922 25486 tablet_replica.cc:330] T dc799afaa39e49c584562742b9477436 P 633cabbcad0347b698839df57695c6cf: stopping tablet replica
I20260807 07:01:38.559967 25484 ts_tablet_manager.cc:1905] T dc799afaa39e49c584562742b9477436 P 62aacfc55612476eac88f5a78966f57e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:01:38.560124 25486 raft_consensus.cc:2229] T dc799afaa39e49c584562742b9477436 P 633cabbcad0347b698839df57695c6cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:01:38.560300 25486 raft_consensus.cc:2258] T dc799afaa39e49c584562742b9477436 P 633cabbcad0347b698839df57695c6cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:01:38.561128 25485 ts_tablet_manager.cc:1905] T dc799afaa39e49c584562742b9477436 P 8b9c6ea9dc714845ad211e5fd0e146cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:01:38.561363 25486 ts_tablet_manager.cc:1905] T dc799afaa39e49c584562742b9477436 P 633cabbcad0347b698839df57695c6cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:01:38.565981 25486 ts_tablet_manager.cc:1918] T dc799afaa39e49c584562742b9477436 P 633cabbcad0347b698839df57695c6cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 07:01:38.566125 25486 log.cc:1196] T dc799afaa39e49c584562742b9477436 P 633cabbcad0347b698839df57695c6cf: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-1/wal/wals/dc799afaa39e49c584562742b9477436
I20260807 07:01:38.566371 25484 ts_tablet_manager.cc:1918] T dc799afaa39e49c584562742b9477436 P 62aacfc55612476eac88f5a78966f57e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 07:01:38.566466 25486 ts_tablet_manager.cc:1939] T dc799afaa39e49c584562742b9477436 P 633cabbcad0347b698839df57695c6cf: Deleting consensus metadata
I20260807 07:01:38.566522 25484 log.cc:1196] T dc799afaa39e49c584562742b9477436 P 62aacfc55612476eac88f5a78966f57e: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-0/wal/wals/dc799afaa39e49c584562742b9477436
I20260807 07:01:38.566972 25484 ts_tablet_manager.cc:1939] T dc799afaa39e49c584562742b9477436 P 62aacfc55612476eac88f5a78966f57e: Deleting consensus metadata
I20260807 07:01:38.567358 24543 catalog_manager.cc:4841] TS 633cabbcad0347b698839df57695c6cf (127.21.83.2:43979): tablet dc799afaa39e49c584562742b9477436 (table test-table [id=4b7211acc4204ea2b3d8ec391c4a5fd9]) successfully deleted
I20260807 07:01:38.567917 25485 ts_tablet_manager.cc:1918] T dc799afaa39e49c584562742b9477436 P 8b9c6ea9dc714845ad211e5fd0e146cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 07:01:38.567909 24538 catalog_manager.cc:4841] TS 62aacfc55612476eac88f5a78966f57e (127.21.83.1:33999): tablet dc799afaa39e49c584562742b9477436 (table test-table [id=4b7211acc4204ea2b3d8ec391c4a5fd9]) successfully deleted
I20260807 07:01:38.568068 25485 log.cc:1196] T dc799afaa39e49c584562742b9477436 P 8b9c6ea9dc714845ad211e5fd0e146cb: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-2/wal/wals/dc799afaa39e49c584562742b9477436
I20260807 07:01:38.568416 25485 ts_tablet_manager.cc:1939] T dc799afaa39e49c584562742b9477436 P 8b9c6ea9dc714845ad211e5fd0e146cb: Deleting consensus metadata
I20260807 07:01:38.569172 24538 catalog_manager.cc:4841] TS 8b9c6ea9dc714845ad211e5fd0e146cb (127.21.83.3:39465): tablet dc799afaa39e49c584562742b9477436 (table test-table [id=4b7211acc4204ea2b3d8ec391c4a5fd9]) successfully deleted
I20260807 07:01:38.572297 24766 tablet_service.cc:1467] Processing CreateTablet for tablet 2fcacaa2c509481e829b9a9d3f934ed4 (DEFAULT_TABLE table=test-table [id=915af781eda840da8d475d73bc8ed9a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:01:38.572543 24996 tablet_service.cc:1467] Processing CreateTablet for tablet 2fcacaa2c509481e829b9a9d3f934ed4 (DEFAULT_TABLE table=test-table [id=915af781eda840da8d475d73bc8ed9a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:01:38.572719 24766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fcacaa2c509481e829b9a9d3f934ed4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:01:38.572933 24996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fcacaa2c509481e829b9a9d3f934ed4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:01:38.573710 25214 tablet_service.cc:1467] Processing CreateTablet for tablet 2fcacaa2c509481e829b9a9d3f934ed4 (DEFAULT_TABLE table=test-table [id=915af781eda840da8d475d73bc8ed9a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:01:38.574025 25214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fcacaa2c509481e829b9a9d3f934ed4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:01:38.577587 25502 tablet_bootstrap.cc:492] T 2fcacaa2c509481e829b9a9d3f934ed4 P 62aacfc55612476eac88f5a78966f57e: Bootstrap starting.
I20260807 07:01:38.579218 25506 tablet_bootstrap.cc:492] T 2fcacaa2c509481e829b9a9d3f934ed4 P 8b9c6ea9dc714845ad211e5fd0e146cb: Bootstrap starting.
I20260807 07:01:38.579384 25502 tablet_bootstrap.cc:654] T 2fcacaa2c509481e829b9a9d3f934ed4 P 62aacfc55612476eac88f5a78966f57e: Neither blocks nor log segments found. Creating new log.
I20260807 07:01:38.580750 25506 tablet_bootstrap.cc:654] T 2fcacaa2c509481e829b9a9d3f934ed4 P 8b9c6ea9dc714845ad211e5fd0e146cb: Neither blocks nor log segments found. Creating new log.
I20260807 07:01:38.581622 25502 tablet_bootstrap.cc:492] T 2fcacaa2c509481e829b9a9d3f934ed4 P 62aacfc55612476eac88f5a78966f57e: No bootstrap required, opened a new log
I20260807 07:01:38.581879 25502 ts_tablet_manager.cc:1397] T 2fcacaa2c509481e829b9a9d3f934ed4 P 62aacfc55612476eac88f5a78966f57e: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20260807 07:01:38.582765 25502 raft_consensus.cc:348] T 2fcacaa2c509481e829b9a9d3f934ed4 P 62aacfc55612476eac88f5a78966f57e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } }
I20260807 07:01:38.583022 25506 tablet_bootstrap.cc:492] T 2fcacaa2c509481e829b9a9d3f934ed4 P 8b9c6ea9dc714845ad211e5fd0e146cb: No bootstrap required, opened a new log
I20260807 07:01:38.583115 25502 raft_consensus.cc:374] T 2fcacaa2c509481e829b9a9d3f934ed4 P 62aacfc55612476eac88f5a78966f57e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:01:38.583206 25506 ts_tablet_manager.cc:1397] T 2fcacaa2c509481e829b9a9d3f934ed4 P 8b9c6ea9dc714845ad211e5fd0e146cb: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260807 07:01:38.583209 25502 raft_consensus.cc:729] T 2fcacaa2c509481e829b9a9d3f934ed4 P 62aacfc55612476eac88f5a78966f57e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62aacfc55612476eac88f5a78966f57e, State: Initialized, Role: FOLLOWER
I20260807 07:01:38.583453 25502 consensus_queue.cc:260] T 2fcacaa2c509481e829b9a9d3f934ed4 P 62aacfc55612476eac88f5a78966f57e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } }
I20260807 07:01:38.584499 25502 ts_tablet_manager.cc:1428] T 2fcacaa2c509481e829b9a9d3f934ed4 P 62aacfc55612476eac88f5a78966f57e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260807 07:01:38.584394 25506 raft_consensus.cc:348] T 2fcacaa2c509481e829b9a9d3f934ed4 P 8b9c6ea9dc714845ad211e5fd0e146cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } }
I20260807 07:01:38.586097 25506 raft_consensus.cc:374] T 2fcacaa2c509481e829b9a9d3f934ed4 P 8b9c6ea9dc714845ad211e5fd0e146cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:01:38.586284 25506 raft_consensus.cc:729] T 2fcacaa2c509481e829b9a9d3f934ed4 P 8b9c6ea9dc714845ad211e5fd0e146cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b9c6ea9dc714845ad211e5fd0e146cb, State: Initialized, Role: FOLLOWER
I20260807 07:01:38.586364 25501 tablet_bootstrap.cc:492] T 2fcacaa2c509481e829b9a9d3f934ed4 P 633cabbcad0347b698839df57695c6cf: Bootstrap starting.
I20260807 07:01:38.586624 25506 consensus_queue.cc:260] T 2fcacaa2c509481e829b9a9d3f934ed4 P 8b9c6ea9dc714845ad211e5fd0e146cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } }
I20260807 07:01:38.587248 25506 ts_tablet_manager.cc:1428] T 2fcacaa2c509481e829b9a9d3f934ed4 P 8b9c6ea9dc714845ad211e5fd0e146cb: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20260807 07:01:38.588224 25501 tablet_bootstrap.cc:654] T 2fcacaa2c509481e829b9a9d3f934ed4 P 633cabbcad0347b698839df57695c6cf: Neither blocks nor log segments found. Creating new log.
I20260807 07:01:38.590546 25501 tablet_bootstrap.cc:492] T 2fcacaa2c509481e829b9a9d3f934ed4 P 633cabbcad0347b698839df57695c6cf: No bootstrap required, opened a new log
I20260807 07:01:38.590711 25501 ts_tablet_manager.cc:1397] T 2fcacaa2c509481e829b9a9d3f934ed4 P 633cabbcad0347b698839df57695c6cf: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260807 07:01:38.591355 25501 raft_consensus.cc:348] T 2fcacaa2c509481e829b9a9d3f934ed4 P 633cabbcad0347b698839df57695c6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } }
I20260807 07:01:38.591567 25501 raft_consensus.cc:374] T 2fcacaa2c509481e829b9a9d3f934ed4 P 633cabbcad0347b698839df57695c6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:01:38.591650 25501 raft_consensus.cc:729] T 2fcacaa2c509481e829b9a9d3f934ed4 P 633cabbcad0347b698839df57695c6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 633cabbcad0347b698839df57695c6cf, State: Initialized, Role: FOLLOWER
I20260807 07:01:38.591874 25501 consensus_queue.cc:260] T 2fcacaa2c509481e829b9a9d3f934ed4 P 633cabbcad0347b698839df57695c6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } }
I20260807 07:01:38.592320 25501 ts_tablet_manager.cc:1428] T 2fcacaa2c509481e829b9a9d3f934ed4 P 633cabbcad0347b698839df57695c6cf: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260807 07:01:38.705490 25363 raft_consensus.cc:482] T 2fcacaa2c509481e829b9a9d3f934ed4 P 633cabbcad0347b698839df57695c6cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:01:38.705703 25363 raft_consensus.cc:504] T 2fcacaa2c509481e829b9a9d3f934ed4 P 633cabbcad0347b698839df57695c6cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } }
I20260807 07:01:38.706322 25363 leader_election.cc:288] T 2fcacaa2c509481e829b9a9d3f934ed4 P 633cabbcad0347b698839df57695c6cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62aacfc55612476eac88f5a78966f57e (127.21.83.1:33999), 8b9c6ea9dc714845ad211e5fd0e146cb (127.21.83.3:39465)
I20260807 07:01:38.706902 24804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fcacaa2c509481e829b9a9d3f934ed4" candidate_uuid: "633cabbcad0347b698839df57695c6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62aacfc55612476eac88f5a78966f57e" is_pre_election: true
I20260807 07:01:38.706924 25247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fcacaa2c509481e829b9a9d3f934ed4" candidate_uuid: "633cabbcad0347b698839df57695c6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" is_pre_election: true
I20260807 07:01:38.707130 25247 raft_consensus.cc:2454] T 2fcacaa2c509481e829b9a9d3f934ed4 P 8b9c6ea9dc714845ad211e5fd0e146cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 633cabbcad0347b698839df57695c6cf in term 0.
I20260807 07:01:38.707136 24804 raft_consensus.cc:2454] T 2fcacaa2c509481e829b9a9d3f934ed4 P 62aacfc55612476eac88f5a78966f57e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 633cabbcad0347b698839df57695c6cf in term 0.
I20260807 07:01:38.707578 24919 leader_election.cc:302] T 2fcacaa2c509481e829b9a9d3f934ed4 P 633cabbcad0347b698839df57695c6cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62aacfc55612476eac88f5a78966f57e, 633cabbcad0347b698839df57695c6cf; no voters:
I20260807 07:01:38.707846 25363 raft_consensus.cc:2781] T 2fcacaa2c509481e829b9a9d3f934ed4 P 633cabbcad0347b698839df57695c6cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 07:01:38.707949 25363 raft_consensus.cc:482] T 2fcacaa2c509481e829b9a9d3f934ed4 P 633cabbcad0347b698839df57695c6cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:01:38.708032 25363 raft_consensus.cc:3037] T 2fcacaa2c509481e829b9a9d3f934ed4 P 633cabbcad0347b698839df57695c6cf [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:01:38.710098 25363 raft_consensus.cc:504] T 2fcacaa2c509481e829b9a9d3f934ed4 P 633cabbcad0347b698839df57695c6cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } }
I20260807 07:01:38.710588 25363 leader_election.cc:288] T 2fcacaa2c509481e829b9a9d3f934ed4 P 633cabbcad0347b698839df57695c6cf [CANDIDATE]: Term 1 election: Requested vote from peers 62aacfc55612476eac88f5a78966f57e (127.21.83.1:33999), 8b9c6ea9dc714845ad211e5fd0e146cb (127.21.83.3:39465)
I20260807 07:01:38.711006 24804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fcacaa2c509481e829b9a9d3f934ed4" candidate_uuid: "633cabbcad0347b698839df57695c6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62aacfc55612476eac88f5a78966f57e"
I20260807 07:01:38.711090 25247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fcacaa2c509481e829b9a9d3f934ed4" candidate_uuid: "633cabbcad0347b698839df57695c6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb"
I20260807 07:01:38.711182 24804 raft_consensus.cc:3037] T 2fcacaa2c509481e829b9a9d3f934ed4 P 62aacfc55612476eac88f5a78966f57e [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:01:38.711297 25247 raft_consensus.cc:3037] T 2fcacaa2c509481e829b9a9d3f934ed4 P 8b9c6ea9dc714845ad211e5fd0e146cb [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:01:38.713189 24804 raft_consensus.cc:2454] T 2fcacaa2c509481e829b9a9d3f934ed4 P 62aacfc55612476eac88f5a78966f57e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 633cabbcad0347b698839df57695c6cf in term 1.
I20260807 07:01:38.713337 25247 raft_consensus.cc:2454] T 2fcacaa2c509481e829b9a9d3f934ed4 P 8b9c6ea9dc714845ad211e5fd0e146cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 633cabbcad0347b698839df57695c6cf in term 1.
I20260807 07:01:38.713570 24919 leader_election.cc:302] T 2fcacaa2c509481e829b9a9d3f934ed4 P 633cabbcad0347b698839df57695c6cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62aacfc55612476eac88f5a78966f57e, 633cabbcad0347b698839df57695c6cf; no voters:
I20260807 07:01:38.713845 25363 raft_consensus.cc:2781] T 2fcacaa2c509481e829b9a9d3f934ed4 P 633cabbcad0347b698839df57695c6cf [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:01:38.713975 25363 raft_consensus.cc:686] T 2fcacaa2c509481e829b9a9d3f934ed4 P 633cabbcad0347b698839df57695c6cf [term 1 LEADER]: Becoming Leader. State: Replica: 633cabbcad0347b698839df57695c6cf, State: Running, Role: LEADER
I20260807 07:01:38.714246 25363 consensus_queue.cc:237] T 2fcacaa2c509481e829b9a9d3f934ed4 P 633cabbcad0347b698839df57695c6cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } }
I20260807 07:01:38.716822 24558 catalog_manager.cc:5495] T 2fcacaa2c509481e829b9a9d3f934ed4 P 633cabbcad0347b698839df57695c6cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 633cabbcad0347b698839df57695c6cf (127.21.83.2). New cstate: current_term: 1 leader_uuid: "633cabbcad0347b698839df57695c6cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "633cabbcad0347b698839df57695c6cf" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 } health_report { overall_health: UNKNOWN } } }
I20260807 07:01:38.754822 25535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:38.743111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51228 (local address 127.21.83.2:43979)
0807 07:01:38.749506 (+ 6395us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:38.749513 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:38.749544 (+ 31us) server_negotiation.cc:407] Connection header received
0807 07:01:38.749639 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:38.749645 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:38.749739 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:38.749873 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:01:38.750382 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:38.751680 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:38.753173 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:38.753613 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:38.753718 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:38.754091 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:01:38.754210 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:01:38.754276 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:38.754370 (+ 94us) server_negotiation.cc:299] Negotiation successful
0807 07:01:38.754440 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6216,"thread_start_us":6059,"threads_started":1}
I20260807 07:01:38.768932 25535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:38.763895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:38471 (local address 127.21.83.2:43979)
0807 07:01:38.764066 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:38.764072 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:38.764504 (+ 432us) server_negotiation.cc:407] Connection header received
0807 07:01:38.764683 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:38.764688 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:38.764750 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:38.764823 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:01:38.765283 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:38.766013 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:38.767352 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:38.767755 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:38.767862 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:38.768217 (+ 355us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:01:38.768321 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:01:38.768394 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:38.768511 (+ 117us) server_negotiation.cc:299] Negotiation successful
0807 07:01:38.768581 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260807 07:01:38.769001 25543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:38.763859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.2:43979 (local address 127.21.83.2:38471)
0807 07:01:38.764409 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:38.764424 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:38.764561 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:38.764843 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:38.764847 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:38.764857 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 07:01:38.765115 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:38.765125 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:38.766139 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:38.766150 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:38.767220 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:38.767233 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:38.767908 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:38.767914 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:38.768034 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:38.768073 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 07:01:38.768405 (+ 332us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 07:01:38.768416 (+ 11us) client_negotiation.cc:769] Sending connection context
0807 07:01:38.768486 (+ 70us) client_negotiation.cc:240] Negotiation successful
0807 07:01:38.768622 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":300,"threads_started":1}
I20260807 07:01:38.777854 25247 raft_consensus.cc:1261] T 2fcacaa2c509481e829b9a9d3f934ed4 P 8b9c6ea9dc714845ad211e5fd0e146cb [term 1 FOLLOWER]: Refusing update from remote peer 633cabbcad0347b698839df57695c6cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:01:38.777918 24804 raft_consensus.cc:1261] T 2fcacaa2c509481e829b9a9d3f934ed4 P 62aacfc55612476eac88f5a78966f57e [term 1 FOLLOWER]: Refusing update from remote peer 633cabbcad0347b698839df57695c6cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:01:38.778466 25363 consensus_queue.cc:1035] T 2fcacaa2c509481e829b9a9d3f934ed4 P 633cabbcad0347b698839df57695c6cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "62aacfc55612476eac88f5a78966f57e" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 33999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:01:38.778772 25527 consensus_queue.cc:1035] T 2fcacaa2c509481e829b9a9d3f934ed4 P 633cabbcad0347b698839df57695c6cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9c6ea9dc714845ad211e5fd0e146cb" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 07:01:38.798497 25449 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260807 07:01:40.375885 24558 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:01:40.376163 24558 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:01:40.380296 24558 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bd7c08a8e2434a539ae91b39b7c750dc: Sending DeleteTablet for 3 replicas of tablet 2fcacaa2c509481e829b9a9d3f934ed4
I20260807 07:01:40.381167 24766 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fcacaa2c509481e829b9a9d3f934ed4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:01:40 UTC) from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:35030
I20260807 07:01:40.381168 25000 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fcacaa2c509481e829b9a9d3f934ed4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:01:40 UTC) from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:42646
I20260807 07:01:40.381301 25214 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fcacaa2c509481e829b9a9d3f934ed4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:01:40 UTC) from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:47534
I20260807 07:01:40.381742 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 24664
I20260807 07:01:40.393638 25581 tablet_replica.cc:330] T 2fcacaa2c509481e829b9a9d3f934ed4 P 633cabbcad0347b698839df57695c6cf: stopping tablet replica
I20260807 07:01:40.394215 25580 tablet_replica.cc:330] T 2fcacaa2c509481e829b9a9d3f934ed4 P 62aacfc55612476eac88f5a78966f57e: stopping tablet replica
I20260807 07:01:40.394593 25581 raft_consensus.cc:2229] T 2fcacaa2c509481e829b9a9d3f934ed4 P 633cabbcad0347b698839df57695c6cf [term 1 LEADER]: Raft consensus shutting down.
I20260807 07:01:40.394883 25580 raft_consensus.cc:2229] T 2fcacaa2c509481e829b9a9d3f934ed4 P 62aacfc55612476eac88f5a78966f57e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:01:40.395540 25581 raft_consensus.cc:2258] T 2fcacaa2c509481e829b9a9d3f934ed4 P 633cabbcad0347b698839df57695c6cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:01:40.395877 25580 raft_consensus.cc:2258] T 2fcacaa2c509481e829b9a9d3f934ed4 P 62aacfc55612476eac88f5a78966f57e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:01:40.397140 25580 ts_tablet_manager.cc:1905] T 2fcacaa2c509481e829b9a9d3f934ed4 P 62aacfc55612476eac88f5a78966f57e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:01:40.399717 25584 tablet_replica.cc:330] T 2fcacaa2c509481e829b9a9d3f934ed4 P 8b9c6ea9dc714845ad211e5fd0e146cb: stopping tablet replica
I20260807 07:01:40.399973 25584 raft_consensus.cc:2229] T 2fcacaa2c509481e829b9a9d3f934ed4 P 8b9c6ea9dc714845ad211e5fd0e146cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:01:40.400741 25581 ts_tablet_manager.cc:1905] T 2fcacaa2c509481e829b9a9d3f934ed4 P 633cabbcad0347b698839df57695c6cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:01:40.405437 25580 ts_tablet_manager.cc:1918] T 2fcacaa2c509481e829b9a9d3f934ed4 P 62aacfc55612476eac88f5a78966f57e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 07:01:40.405582 25580 log.cc:1196] T 2fcacaa2c509481e829b9a9d3f934ed4 P 62aacfc55612476eac88f5a78966f57e: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-0/wal/wals/2fcacaa2c509481e829b9a9d3f934ed4
I20260807 07:01:40.406024 25580 ts_tablet_manager.cc:1939] T 2fcacaa2c509481e829b9a9d3f934ed4 P 62aacfc55612476eac88f5a78966f57e: Deleting consensus metadata
I20260807 07:01:40.406121 25581 ts_tablet_manager.cc:1918] T 2fcacaa2c509481e829b9a9d3f934ed4 P 633cabbcad0347b698839df57695c6cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 07:01:40.406435 25584 raft_consensus.cc:2258] T 2fcacaa2c509481e829b9a9d3f934ed4 P 8b9c6ea9dc714845ad211e5fd0e146cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:01:40.406759 25581 log.cc:1196] T 2fcacaa2c509481e829b9a9d3f934ed4 P 633cabbcad0347b698839df57695c6cf: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-1/wal/wals/2fcacaa2c509481e829b9a9d3f934ed4
I20260807 07:01:40.407563 25584 ts_tablet_manager.cc:1905] T 2fcacaa2c509481e829b9a9d3f934ed4 P 8b9c6ea9dc714845ad211e5fd0e146cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:01:40.407924 24538 catalog_manager.cc:4841] TS 62aacfc55612476eac88f5a78966f57e (127.21.83.1:33999): tablet 2fcacaa2c509481e829b9a9d3f934ed4 (table test-table [id=915af781eda840da8d475d73bc8ed9a4]) successfully deleted
I20260807 07:01:40.408293 25581 ts_tablet_manager.cc:1939] T 2fcacaa2c509481e829b9a9d3f934ed4 P 633cabbcad0347b698839df57695c6cf: Deleting consensus metadata
Aug 07 07:01:40 dist-test-slave-ncp9 krb5kdc[24491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086098, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.83.1@KRBTEST.COM
I20260807 07:01:40.409338 24543 catalog_manager.cc:4841] TS 633cabbcad0347b698839df57695c6cf (127.21.83.2:43979): tablet 2fcacaa2c509481e829b9a9d3f934ed4 (table test-table [id=915af781eda840da8d475d73bc8ed9a4]) successfully deleted
I20260807 07:01:40.412873 25584 ts_tablet_manager.cc:1918] T 2fcacaa2c509481e829b9a9d3f934ed4 P 8b9c6ea9dc714845ad211e5fd0e146cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 07:01:40.413025 25584 log.cc:1196] T 2fcacaa2c509481e829b9a9d3f934ed4 P 8b9c6ea9dc714845ad211e5fd0e146cb: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786086078080656-21836-0/minicluster-data/ts-2/wal/wals/2fcacaa2c509481e829b9a9d3f934ed4
I20260807 07:01:40.413394 25584 ts_tablet_manager.cc:1939] T 2fcacaa2c509481e829b9a9d3f934ed4 P 8b9c6ea9dc714845ad211e5fd0e146cb: Deleting consensus metadata
I20260807 07:01:40.414254 24538 catalog_manager.cc:4841] TS 8b9c6ea9dc714845ad211e5fd0e146cb (127.21.83.3:39465): tablet 2fcacaa2c509481e829b9a9d3f934ed4 (table test-table [id=915af781eda840da8d475d73bc8ed9a4]) successfully deleted
I20260807 07:01:40.416527 25587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:40.383203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43820 (local address 127.21.83.1:33999)
0807 07:01:40.394406 (+ 11203us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:40.394413 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:40.394851 (+ 438us) server_negotiation.cc:407] Connection header received
0807 07:01:40.395025 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:40.395029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:40.395096 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:40.395192 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:40.396689 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:40.397924 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:40.399843 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:40.400445 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:40.400593 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:40.409981 (+ 9388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:40.409999 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:40.410020 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:40.410063 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:40.413820 (+ 3757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:40.414585 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:40.414595 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:40.414599 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:40.414775 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:40.415096 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:40.415101 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:40.415105 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:40.415504 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:40.415825 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:40.415955 (+ 130us) server_negotiation.cc:299] Negotiation successful
0807 07:01:40.416163 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11053,"thread_start_us":10919,"threads_started":1}
W20260807 07:01:40.417165 24876 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43820
W20260807 07:01:40.417728 21836 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 24664: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 07:01:40.417945 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 24664
I20260807 07:01:40.444380 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 24892
Aug 07 07:01:40 dist-test-slave-ncp9 krb5kdc[24491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086098, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.83.2@KRBTEST.COM
I20260807 07:01:40.458633 25588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:40.445082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51236 (local address 127.21.83.2:43979)
0807 07:01:40.445652 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:40.445656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:40.445673 (+ 17us) server_negotiation.cc:407] Connection header received
0807 07:01:40.445724 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:40.445727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:40.445789 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:40.445875 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:40.447022 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:40.447763 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:40.448944 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:40.449315 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:40.449404 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:40.453391 (+ 3987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:40.453404 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:40.453418 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:40.453467 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:40.456025 (+ 2558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:40.456654 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:40.456662 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:40.456664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:40.456796 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:40.457217 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:40.457221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:40.457223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:40.457549 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:40.457842 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:40.458113 (+ 271us) server_negotiation.cc:299] Negotiation successful
0807 07:01:40.458318 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":306,"threads_started":1}
W20260807 07:01:40.459291 25105 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51236
W20260807 07:01:40.459848 21836 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 24892: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 07:01:40.459969 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 24892
I20260807 07:01:40.491215 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 25118
Aug 07 07:01:40 dist-test-slave-ncp9 krb5kdc[24491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086098, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.83.3@KRBTEST.COM
I20260807 07:01:40.507180 25589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:40.491919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41942 (local address 127.21.83.3:39465)
0807 07:01:40.492404 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:40.492409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:40.492427 (+ 18us) server_negotiation.cc:407] Connection header received
0807 07:01:40.492483 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:40.492486 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:40.492559 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:40.492653 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:40.494033 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:40.494823 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:40.496079 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:40.496502 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:40.496608 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:40.501162 (+ 4554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:40.501182 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:40.501199 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:40.501248 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:40.504182 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:40.505014 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:40.505022 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:40.505024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:40.505193 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:40.505582 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:40.505586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:40.505588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:40.505954 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:40.506279 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:40.506600 (+ 321us) server_negotiation.cc:299] Negotiation successful
0807 07:01:40.506844 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":254,"threads_started":1}
W20260807 07:01:40.507905 25320 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41942
W20260807 07:01:40.508519 21836 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 25118: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 07:01:40.508694 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 25118
I20260807 07:01:40.532608 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 24511
I20260807 07:01:40.533880 24622 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:01:40.649906 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 24511
2026-08-07T07:01:40Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8770 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 07:01:40 dist-test-slave-ncp9 krb5kdc[25594](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 07:01:40 dist-test-slave-ncp9 krb5kdc[25594](info): set up 2 sockets
Aug 07 07:01:40 dist-test-slave-ncp9 krb5kdc[25594](info): commencing operation
krb5kdc: starting...
W20260807 07:01:41.847060 21836 mini_kdc.cc:121] Time spent starting KDC: real 1.149s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Aug 07 07:01:41 dist-test-slave-ncp9 krb5kdc[25594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T07:01:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:01:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
I20260807 07:01:42.083819 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:35255
--webserver_interface=127.21.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:38957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:35255
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:01:42.418347 25620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:01:42.418651 25620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:01:42.418753 25620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:01:42.426039 25620 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:01:42.426152 25620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:01:42.426208 25620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:01:42.426265 25620 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:01:42.426312 25620 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:01:42.436511 25620 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:38957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:35255
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:35255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:01:42.437927 25620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:01:42.439217 25620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:01:42.451784 25628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:42.451766 25629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:42.451879 25634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:01:42.452431 25620 server_base.cc:1034] running on GCE node
I20260807 07:01:42.453249 25620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:01:42.454644 25620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:01:42.455847 25620 hybrid_clock.cc:648] HybridClock initialized: now 1786086102455794 us; error 38 us; skew 500 ppm
I20260807 07:01:42.456171 25620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:01:42 dist-test-slave-ncp9 krb5kdc[25594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086102, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:01:42.459275 25620 init.cc:377] Logged in from keytab as kudu/127.21.83.62@KRBTEST.COM (short username kudu)
I20260807 07:01:42.460817 25620 webserver.cc:460] Webserver started at http://127.21.83.62:44109/ using document root <none> and password file <none>
I20260807 07:01:42.461567 25620 fs_manager.cc:359] Metadata directory not provided
I20260807 07:01:42.461695 25620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:01:42.462007 25620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:01:42.465296 25620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "4ab251adcbb6458b8b825e8089b27d1f"
format_stamp: "Formatted at 2026-08-07 07:01:42 on dist-test-slave-ncp9"
server_key: "82d317563f83523b86047e60c5ad7780"
server_key_iv: "f5ffa23898dc9ae8629d9845ac171d0b"
server_key_version: "encryptionkey@0"
I20260807 07:01:42.466127 25620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "4ab251adcbb6458b8b825e8089b27d1f"
format_stamp: "Formatted at 2026-08-07 07:01:42 on dist-test-slave-ncp9"
server_key: "82d317563f83523b86047e60c5ad7780"
server_key_iv: "f5ffa23898dc9ae8629d9845ac171d0b"
server_key_version: "encryptionkey@0"
I20260807 07:01:42.471598 25620 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260807 07:01:42.475080 25644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:42.476342 25620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 07:01:42.476507 25620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "4ab251adcbb6458b8b825e8089b27d1f"
format_stamp: "Formatted at 2026-08-07 07:01:42 on dist-test-slave-ncp9"
server_key: "82d317563f83523b86047e60c5ad7780"
server_key_iv: "f5ffa23898dc9ae8629d9845ac171d0b"
server_key_version: "encryptionkey@0"
I20260807 07:01:42.476707 25620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:01:42.497337 25620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:01:42.501394 25620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:01:42.501659 25620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:01:42.519019 25620 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.62:35255
I20260807 07:01:42.519038 25717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.62:35255 every 8 connection(s)
I20260807 07:01:42.520390 25620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:01:42.523940 25719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:01:42.527115 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 25620
I20260807 07:01:42.527328 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:01:42.527953 21836 external_mini_cluster.cc:1428] Setting key a8f93d7c15a97811ac2e544aef875daa
I20260807 07:01:42.536301 25719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ab251adcbb6458b8b825e8089b27d1f: Bootstrap starting.
I20260807 07:01:42.540181 25719 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ab251adcbb6458b8b825e8089b27d1f: Neither blocks nor log segments found. Creating new log.
I20260807 07:01:42.541204 25719 log.cc:824] T 00000000000000000000000000000000 P 4ab251adcbb6458b8b825e8089b27d1f: Log is configured to *not* fsync() on all Append() calls
Aug 07 07:01:42 dist-test-slave-ncp9 krb5kdc[25594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:01:42.544097 25719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ab251adcbb6458b8b825e8089b27d1f: No bootstrap required, opened a new log
I20260807 07:01:42.550356 25724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:42.529963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35732 (local address 127.21.83.62:35255)
0807 07:01:42.530657 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:42.530668 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:42.530708 (+ 40us) server_negotiation.cc:407] Connection header received
0807 07:01:42.532321 (+ 1613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:42.532342 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:42.532709 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:42.533054 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:42.534784 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:42.536043 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:42.537863 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:42.538926 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:42.539097 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:42.543321 (+ 4224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:42.543329 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:42.543343 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:42.543377 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:42.546466 (+ 3089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:42.547229 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:42.547235 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:42.547245 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:42.547418 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:42.547793 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:42.547796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:42.547798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:42.548293 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:42.548521 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:42.548995 (+ 474us) server_negotiation.cc:299] Negotiation successful
0807 07:01:42.549359 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":306,"threads_started":1}
I20260807 07:01:42.552687 25719 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ab251adcbb6458b8b825e8089b27d1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ab251adcbb6458b8b825e8089b27d1f" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 35255 } }
I20260807 07:01:42.553187 25719 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ab251adcbb6458b8b825e8089b27d1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:01:42.553296 25719 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ab251adcbb6458b8b825e8089b27d1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ab251adcbb6458b8b825e8089b27d1f, State: Initialized, Role: FOLLOWER
I20260807 07:01:42.553876 25719 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ab251adcbb6458b8b825e8089b27d1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ab251adcbb6458b8b825e8089b27d1f" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 35255 } }
I20260807 07:01:42.554122 25719 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ab251adcbb6458b8b825e8089b27d1f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:01:42.554247 25719 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ab251adcbb6458b8b825e8089b27d1f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:01:42.554399 25719 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ab251adcbb6458b8b825e8089b27d1f [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:01:42.556634 25719 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ab251adcbb6458b8b825e8089b27d1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ab251adcbb6458b8b825e8089b27d1f" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 35255 } }
I20260807 07:01:42.557246 25719 leader_election.cc:302] T 00000000000000000000000000000000 P 4ab251adcbb6458b8b825e8089b27d1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ab251adcbb6458b8b825e8089b27d1f; no voters:
I20260807 07:01:42.557893 25719 leader_election.cc:288] T 00000000000000000000000000000000 P 4ab251adcbb6458b8b825e8089b27d1f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 07:01:42.558033 25726 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ab251adcbb6458b8b825e8089b27d1f [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:01:42.559048 25726 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ab251adcbb6458b8b825e8089b27d1f [term 1 LEADER]: Becoming Leader. State: Replica: 4ab251adcbb6458b8b825e8089b27d1f, State: Running, Role: LEADER
I20260807 07:01:42.559605 25726 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ab251adcbb6458b8b825e8089b27d1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ab251adcbb6458b8b825e8089b27d1f" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 35255 } }
I20260807 07:01:42.559986 25719 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ab251adcbb6458b8b825e8089b27d1f [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:01:42.562376 25730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ab251adcbb6458b8b825e8089b27d1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ab251adcbb6458b8b825e8089b27d1f. Latest consensus state: current_term: 1 leader_uuid: "4ab251adcbb6458b8b825e8089b27d1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ab251adcbb6458b8b825e8089b27d1f" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 35255 } } }
I20260807 07:01:42.562884 25730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ab251adcbb6458b8b825e8089b27d1f [sys.catalog]: This master's current role is: LEADER
I20260807 07:01:42.562884 25729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ab251adcbb6458b8b825e8089b27d1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ab251adcbb6458b8b825e8089b27d1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ab251adcbb6458b8b825e8089b27d1f" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 35255 } } }
I20260807 07:01:42.563293 25729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ab251adcbb6458b8b825e8089b27d1f [sys.catalog]: This master's current role is: LEADER
I20260807 07:01:42.565300 25734 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:01:42.571213 25734 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:01:42.580037 25734 catalog_manager.cc:1347] Generated new cluster ID: 2badb3592615489aab7db9d8fb03d102
I20260807 07:01:42.580163 25734 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:01:42.598937 25734 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:01:42.600715 25734 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:01:42.625726 25734 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ab251adcbb6458b8b825e8089b27d1f: Generated new TSK 0
I20260807 07:01:42.626585 25734 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.83.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab.
Entry for principal kudu/127.21.83.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab.
I20260807 07:01:42.682493 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.1:0
--local_ip_for_outbound_sockets=127.21.83.1
--webserver_interface=127.21.83.1
--webserver_port=0
--tserver_master_addrs=127.21.83.62:35255
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab
--principal=kudu/127.21.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:38957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:01:43.008431 25757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:01:43.008746 25757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:01:43.008867 25757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:01:43.016227 25757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:01:43.016347 25757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:01:43.016467 25757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.1
I20260807 07:01:43.026448 25757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:38957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab
--principal=kudu/127.21.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.83.1
--webserver_port=0
--tserver_master_addrs=127.21.83.62:35255
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.1
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:01:43.027717 25757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:01:43.028987 25757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:01:43.041571 25764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:43.041631 25765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:43.041589 25768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:01:43.042742 25757 server_base.cc:1034] running on GCE node
I20260807 07:01:43.043406 25757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:01:43.044368 25757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:01:43.045563 25757 hybrid_clock.cc:648] HybridClock initialized: now 1786086103045514 us; error 43 us; skew 500 ppm
I20260807 07:01:43.045884 25757 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:01:43 dist-test-slave-ncp9 krb5kdc[25594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086103, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:01:43.049144 25757 init.cc:377] Logged in from keytab as kudu/127.21.83.1@KRBTEST.COM (short username kudu)
I20260807 07:01:43.050921 25757 webserver.cc:460] Webserver started at http://127.21.83.1:38419/ using document root <none> and password file <none>
I20260807 07:01:43.051663 25757 fs_manager.cc:359] Metadata directory not provided
I20260807 07:01:43.051811 25757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:01:43.052124 25757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:01:43.055202 25757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-0/data/instance:
uuid: "4316b7ac4ba74241b5a256803011abb8"
format_stamp: "Formatted at 2026-08-07 07:01:43 on dist-test-slave-ncp9"
server_key: "b2b00854401709b6222f9a1dca8f4b15"
server_key_iv: "85766d397962070786e965d9382699ed"
server_key_version: "encryptionkey@0"
I20260807 07:01:43.056048 25757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-0/wal/instance:
uuid: "4316b7ac4ba74241b5a256803011abb8"
format_stamp: "Formatted at 2026-08-07 07:01:43 on dist-test-slave-ncp9"
server_key: "b2b00854401709b6222f9a1dca8f4b15"
server_key_iv: "85766d397962070786e965d9382699ed"
server_key_version: "encryptionkey@0"
I20260807 07:01:43.061936 25757 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 07:01:43.065804 25776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:43.067251 25757 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260807 07:01:43.067428 25757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-0/wal
uuid: "4316b7ac4ba74241b5a256803011abb8"
format_stamp: "Formatted at 2026-08-07 07:01:43 on dist-test-slave-ncp9"
server_key: "b2b00854401709b6222f9a1dca8f4b15"
server_key_iv: "85766d397962070786e965d9382699ed"
server_key_version: "encryptionkey@0"
I20260807 07:01:43.067641 25757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:01:43.088523 25757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:01:43.092375 25757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:01:43.092635 25757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:01:43.093645 25757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:01:43.094995 25757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:01:43.095086 25757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:43.095206 25757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:01:43.095258 25757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:43.125025 25757 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.1:37857
I20260807 07:01:43.125082 25949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.1:37857 every 8 connection(s)
I20260807 07:01:43.126281 25757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
I20260807 07:01:43.132416 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 25757
I20260807 07:01:43.132660 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-0/wal/instance
I20260807 07:01:43.133468 21836 external_mini_cluster.cc:1428] Setting key 989a227e6a3d239c0805b037e0a5613f
Aug 07 07:01:43 dist-test-slave-ncp9 krb5kdc[25594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086103, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.1@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:01:43.146149 25955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:43.128629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:41205 (local address 127.21.83.62:35255)
0807 07:01:43.129076 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:43.129083 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:43.130145 (+ 1062us) server_negotiation.cc:407] Connection header received
0807 07:01:43.131904 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:43.131908 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:43.131989 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:43.132090 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:43.134158 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:43.134958 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:43.136577 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:43.137240 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:43.137368 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:43.140871 (+ 3503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:43.140879 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:43.140883 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:43.140950 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:43.143215 (+ 2265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:43.143958 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:43.143964 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:43.143967 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:43.144115 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:43.144466 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:43.144470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:43.144474 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:43.144802 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:43.144963 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:43.145589 (+ 626us) server_negotiation.cc:299] Negotiation successful
0807 07:01:43.145815 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":192,"threads_started":1}
I20260807 07:01:43.146826 25956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:43.129215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:35255 (local address 127.21.83.1:41205)
0807 07:01:43.129978 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:43.130021 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:43.131637 (+ 1616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:43.132401 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:43.132413 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:43.133180 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:01:43.133935 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:43.133960 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:43.135119 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:43.135124 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:43.136431 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:43.136444 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:43.137439 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:43.137444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:43.137623 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:43.138245 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:01:43.138270 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:01:43.140553 (+ 2283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:01:43.143356 (+ 2803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:43.143365 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:43.143380 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:43.143851 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:43.144205 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:43.144208 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:43.144210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:43.144371 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:43.144948 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:01:43.144954 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:01:43.145243 (+ 289us) client_negotiation.cc:769] Sending connection context
0807 07:01:43.145461 (+ 218us) client_negotiation.cc:240] Negotiation successful
0807 07:01:43.145813 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":338,"threads_started":1}
I20260807 07:01:43.148195 25953 heartbeater.cc:347] Connected to a master server at 127.21.83.62:35255
I20260807 07:01:43.148531 25953 heartbeater.cc:464] Registering TS with master...
I20260807 07:01:43.149354 25953 heartbeater.cc:511] Master 127.21.83.62:35255 requested a full tablet report, sending...
I20260807 07:01:43.151561 25667 ts_manager.cc:194] Registered new tserver with Master: 4316b7ac4ba74241b5a256803011abb8 (127.21.83.1:37857)
I20260807 07:01:43.153054 25667 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.1@KRBTEST.COM'} at 127.21.83.1:41205
WARNING: no policy specified for kudu/127.21.83.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab.
Entry for principal kudu/127.21.83.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab.
I20260807 07:01:43.183560 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.2:0
--local_ip_for_outbound_sockets=127.21.83.2
--webserver_interface=127.21.83.2
--webserver_port=0
--tserver_master_addrs=127.21.83.62:35255
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab
--principal=kudu/127.21.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:38957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:01:43.514423 25962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:01:43.514737 25962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:01:43.514853 25962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:01:43.521968 25962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:01:43.522066 25962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:01:43.522233 25962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.2
I20260807 07:01:43.531569 25962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:38957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab
--principal=kudu/127.21.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.83.2
--webserver_port=0
--tserver_master_addrs=127.21.83.62:35255
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.2
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:01:43.532642 25962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:01:43.533905 25962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:01:43.547052 25968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:43.547094 25967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:43.547115 25971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:01:43.547701 25962 server_base.cc:1034] running on GCE node
I20260807 07:01:43.548202 25962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:01:43.549036 25962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:01:43.550230 25962 hybrid_clock.cc:648] HybridClock initialized: now 1786086103550193 us; error 30 us; skew 500 ppm
I20260807 07:01:43.550583 25962 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:01:43 dist-test-slave-ncp9 krb5kdc[25594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086103, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:01:43.553524 25962 init.cc:377] Logged in from keytab as kudu/127.21.83.2@KRBTEST.COM (short username kudu)
I20260807 07:01:43.554788 25962 webserver.cc:460] Webserver started at http://127.21.83.2:44979/ using document root <none> and password file <none>
I20260807 07:01:43.555409 25962 fs_manager.cc:359] Metadata directory not provided
I20260807 07:01:43.555514 25962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:01:43.555795 25962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:01:43.558544 25962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-1/data/instance:
uuid: "6fe59a29dfd84ebd9ffe2ca83d245161"
format_stamp: "Formatted at 2026-08-07 07:01:43 on dist-test-slave-ncp9"
server_key: "056230c220fcc82f26c5878bd354071b"
server_key_iv: "1ecd095fcdfb380d89f700c2b4f763f0"
server_key_version: "encryptionkey@0"
I20260807 07:01:43.559288 25962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-1/wal/instance:
uuid: "6fe59a29dfd84ebd9ffe2ca83d245161"
format_stamp: "Formatted at 2026-08-07 07:01:43 on dist-test-slave-ncp9"
server_key: "056230c220fcc82f26c5878bd354071b"
server_key_iv: "1ecd095fcdfb380d89f700c2b4f763f0"
server_key_version: "encryptionkey@0"
I20260807 07:01:43.564000 25962 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260807 07:01:43.567348 25982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:43.568603 25962 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 07:01:43.568756 25962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-1/wal
uuid: "6fe59a29dfd84ebd9ffe2ca83d245161"
format_stamp: "Formatted at 2026-08-07 07:01:43 on dist-test-slave-ncp9"
server_key: "056230c220fcc82f26c5878bd354071b"
server_key_iv: "1ecd095fcdfb380d89f700c2b4f763f0"
server_key_version: "encryptionkey@0"
I20260807 07:01:43.568988 25962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:01:43.585747 25962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:01:43.590098 25962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:01:43.590395 25962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:01:43.591423 25962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:01:43.592875 25962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:01:43.592991 25962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260807 07:01:43.593132 25962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:01:43.593187 25962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:43.625186 25962 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.2:38153
I20260807 07:01:43.625223 26162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.2:38153 every 8 connection(s)
I20260807 07:01:43.626492 25962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
I20260807 07:01:43.637020 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 25962
I20260807 07:01:43.637244 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-1/wal/instance
I20260807 07:01:43.637820 21836 external_mini_cluster.cc:1428] Setting key 2f481ae80ad6e2050cefada1f97e2d31
Aug 07 07:01:43 dist-test-slave-ncp9 krb5kdc[25594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086103, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.2@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:01:43.648654 25955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:43.629063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:39229 (local address 127.21.83.62:35255)
0807 07:01:43.629299 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:43.629305 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:43.630530 (+ 1225us) server_negotiation.cc:407] Connection header received
0807 07:01:43.632417 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:43.632422 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:43.632510 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:43.632604 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:43.634291 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:43.635006 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:43.636713 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:43.637311 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:43.637435 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:43.642233 (+ 4798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:43.642239 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:43.642243 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:43.642281 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:43.644966 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:43.646049 (+ 1083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:43.646054 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:43.646057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:43.646221 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:43.646758 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:43.646761 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:43.646764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:43.647072 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:43.647189 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:43.648161 (+ 972us) server_negotiation.cc:299] Negotiation successful
0807 07:01:43.648385 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260807 07:01:43.649783 26166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:43.629537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:35255 (local address 127.21.83.2:39229)
0807 07:01:43.630354 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:43.630401 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:43.632129 (+ 1728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:43.632874 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:43.632884 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:43.633500 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:01:43.634113 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:43.634131 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:43.635184 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:43.635188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:43.636501 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:43.636519 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:43.637565 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:43.637573 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:43.637898 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:43.638762 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:01:43.638795 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:01:43.641913 (+ 3118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:01:43.645153 (+ 3240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:43.645171 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:43.645193 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:43.645869 (+ 676us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:43.646340 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:43.646344 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:43.646348 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:43.646625 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:43.647205 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:01:43.647213 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:01:43.647675 (+ 462us) client_negotiation.cc:769] Sending connection context
0807 07:01:43.647992 (+ 317us) client_negotiation.cc:240] Negotiation successful
0807 07:01:43.648567 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":336,"threads_started":1}
I20260807 07:01:43.651367 26164 heartbeater.cc:347] Connected to a master server at 127.21.83.62:35255
I20260807 07:01:43.651721 26164 heartbeater.cc:464] Registering TS with master...
I20260807 07:01:43.652690 26164 heartbeater.cc:511] Master 127.21.83.62:35255 requested a full tablet report, sending...
I20260807 07:01:43.654577 25667 ts_manager.cc:194] Registered new tserver with Master: 6fe59a29dfd84ebd9ffe2ca83d245161 (127.21.83.2:38153)
I20260807 07:01:43.655653 25667 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.2@KRBTEST.COM'} at 127.21.83.2:39229
WARNING: no policy specified for kudu/127.21.83.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab.
Entry for principal kudu/127.21.83.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab.
I20260807 07:01:43.690021 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.3:0
--local_ip_for_outbound_sockets=127.21.83.3
--webserver_interface=127.21.83.3
--webserver_port=0
--tserver_master_addrs=127.21.83.62:35255
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab
--principal=kudu/127.21.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:38957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:01:44.024125 26169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:01:44.024411 26169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:01:44.024542 26169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:01:44.031749 26169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:01:44.031876 26169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:01:44.032037 26169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.3
I20260807 07:01:44.042120 26169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:38957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab
--principal=kudu/127.21.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.83.3
--webserver_port=0
--tserver_master_addrs=127.21.83.62:35255
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.3
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:01:44.043397 26169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:01:44.044651 26169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:01:44.060985 26179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:44.061765 26178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:44.061923 26181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:01:44.062526 26169 server_base.cc:1034] running on GCE node
I20260807 07:01:44.063265 26169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:01:44.064353 26169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:01:44.065577 26169 hybrid_clock.cc:648] HybridClock initialized: now 1786086104065481 us; error 92 us; skew 500 ppm
I20260807 07:01:44.065955 26169 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:01:44 dist-test-slave-ncp9 krb5kdc[25594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086104, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:01:44.069386 26169 init.cc:377] Logged in from keytab as kudu/127.21.83.3@KRBTEST.COM (short username kudu)
I20260807 07:01:44.071055 26169 webserver.cc:460] Webserver started at http://127.21.83.3:44929/ using document root <none> and password file <none>
I20260807 07:01:44.071810 26169 fs_manager.cc:359] Metadata directory not provided
I20260807 07:01:44.071933 26169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:01:44.072235 26169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:01:44.075282 26169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-2/data/instance:
uuid: "2352da0420bb4176acbd6d23adfbf451"
format_stamp: "Formatted at 2026-08-07 07:01:44 on dist-test-slave-ncp9"
server_key: "c0f595f85f7b52834ab34ea3595c6d9b"
server_key_iv: "ac03a2ec131118192f829ffe897e2391"
server_key_version: "encryptionkey@0"
I20260807 07:01:44.076057 26169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-2/wal/instance:
uuid: "2352da0420bb4176acbd6d23adfbf451"
format_stamp: "Formatted at 2026-08-07 07:01:44 on dist-test-slave-ncp9"
server_key: "c0f595f85f7b52834ab34ea3595c6d9b"
server_key_iv: "ac03a2ec131118192f829ffe897e2391"
server_key_version: "encryptionkey@0"
I20260807 07:01:44.081239 26169 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 07:01:44.084654 26188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:44.085981 26169 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 07:01:44.086138 26169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-2/wal
uuid: "2352da0420bb4176acbd6d23adfbf451"
format_stamp: "Formatted at 2026-08-07 07:01:44 on dist-test-slave-ncp9"
server_key: "c0f595f85f7b52834ab34ea3595c6d9b"
server_key_iv: "ac03a2ec131118192f829ffe897e2391"
server_key_version: "encryptionkey@0"
I20260807 07:01:44.086323 26169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:01:44.104990 26169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:01:44.110494 26169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:01:44.110770 26169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:01:44.111788 26169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:01:44.113147 26169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:01:44.113221 26169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:44.113336 26169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:01:44.113400 26169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:44.154914 26169 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.3:34593
I20260807 07:01:44.154948 26358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.3:34593 every 8 connection(s)
I20260807 07:01:44.156167 26169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
I20260807 07:01:44.156328 25953 heartbeater.cc:503] Master 127.21.83.62:35255 was elected leader, sending a full tablet report...
I20260807 07:01:44.162539 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 26169
I20260807 07:01:44.162796 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-2/wal/instance
I20260807 07:01:44.163277 21836 external_mini_cluster.cc:1428] Setting key eadfbfd2755178a960996489737647b1
Aug 07 07:01:44 dist-test-slave-ncp9 krb5kdc[25594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086104, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.3@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:01:44.178192 26361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:44.158870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:54909 (local address 127.21.83.62:35255)
0807 07:01:44.159323 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:44.159330 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:44.160552 (+ 1222us) server_negotiation.cc:407] Connection header received
0807 07:01:44.162583 (+ 2031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:44.162589 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:44.162702 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:44.162831 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:44.165030 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:44.165953 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:44.167769 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:44.168199 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:44.168299 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:44.171918 (+ 3619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:44.171924 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:44.171927 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:44.171978 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:44.174781 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:44.175609 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:44.175613 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:44.175616 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:44.175747 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:44.176205 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:44.176210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:44.176212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:44.176515 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:44.176643 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:44.177571 (+ 928us) server_negotiation.cc:299] Negotiation successful
0807 07:01:44.177797 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":205,"threads_started":1}
I20260807 07:01:44.179387 26363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:44.159509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:35255 (local address 127.21.83.3:54909)
0807 07:01:44.160397 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:44.160447 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:44.162279 (+ 1832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:44.163103 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:44.163113 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:44.163962 (+ 849us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:01:44.164817 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:44.164840 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:44.166142 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:44.166148 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:44.167635 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:44.167649 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:44.168384 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:44.168389 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:44.168609 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:44.169274 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:01:44.169299 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:01:44.171644 (+ 2345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:01:44.174952 (+ 3308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:44.174960 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:44.174977 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:44.175471 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:44.175827 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:44.175830 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:44.175832 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:44.176056 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:44.176620 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:01:44.176626 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:01:44.177151 (+ 525us) client_negotiation.cc:769] Sending connection context
0807 07:01:44.177418 (+ 267us) client_negotiation.cc:240] Negotiation successful
0807 07:01:44.177968 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":343,"threads_started":1}
I20260807 07:01:44.180958 26359 heartbeater.cc:347] Connected to a master server at 127.21.83.62:35255
I20260807 07:01:44.181283 26359 heartbeater.cc:464] Registering TS with master...
I20260807 07:01:44.182068 26359 heartbeater.cc:511] Master 127.21.83.62:35255 requested a full tablet report, sending...
I20260807 07:01:44.183667 25667 ts_manager.cc:194] Registered new tserver with Master: 2352da0420bb4176acbd6d23adfbf451 (127.21.83.3:34593)
I20260807 07:01:44.184494 25667 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.3@KRBTEST.COM'} at 127.21.83.3:54909
I20260807 07:01:44.191998 21836 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260807 07:01:44.226908 26361 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:01:44.224262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35746 (local address 127.21.83.62:35255)
0807 07:01:44.224431 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:44.224437 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:44.225042 (+ 605us) server_negotiation.cc:407] Connection header received
0807 07:01:44.225197 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:44.225201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:44.225281 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:44.225371 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:44.226714 (+ 1343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35746: BlockingRecv error: recv got EOF from 127.0.0.1:35746 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 07:01:44.229975 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 25757
W20260807 07:01:44.233129 21836 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 25757: Not authorized: Client connection negotiation failed: client connection to 127.21.83.1:37857: server requires authentication, but client does not have Kerberos credentials available
I20260807 07:01:44.233327 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 25757
I20260807 07:01:44.258628 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 25962
W20260807 07:01:44.261631 21836 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 25962: Not authorized: Client connection negotiation failed: client connection to 127.21.83.2:38153: server requires authentication, but client does not have Kerberos credentials available
I20260807 07:01:44.261835 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 25962
I20260807 07:01:44.289453 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 26169
W20260807 07:01:44.292577 21836 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 26169: Not authorized: Client connection negotiation failed: client connection to 127.21.83.3:34593: server requires authentication, but client does not have Kerberos credentials available
I20260807 07:01:44.292721 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 26169
I20260807 07:01:44.316085 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 25620
I20260807 07:01:44.317329 25711 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:01:44.425881 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 25620
2026-08-07T07:01:44Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3770 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786086078080656-21836-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 07:01:44 dist-test-slave-ncp9 krb5kdc[26385](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 07:01:44 dist-test-slave-ncp9 krb5kdc[26385](info): set up 2 sockets
Aug 07 07:01:44 dist-test-slave-ncp9 krb5kdc[26385](info): commencing operation
krb5kdc: starting...
W20260807 07:01:45.609967 21836 mini_kdc.cc:121] Time spent starting KDC: real 1.141s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Aug 07 07:01:45 dist-test-slave-ncp9 krb5kdc[26385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T07:01:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:01:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
I20260807 07:01:45.847954 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:43863
--webserver_interface=127.21.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:45187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:43863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:01:46.177342 26405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:01:46.177688 26405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:01:46.177784 26405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:01:46.185540 26405 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:01:46.185678 26405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:01:46.185725 26405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:01:46.185784 26405 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:01:46.185838 26405 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:01:46.196105 26405 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:45187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:43863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:43863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:01:46.197413 26405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:01:46.198639 26405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:01:46.209986 26410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:46.210033 26411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:46.210126 26414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:01:46.211051 26405 server_base.cc:1034] running on GCE node
I20260807 07:01:46.211791 26405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:01:46.213135 26405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:01:46.214339 26405 hybrid_clock.cc:648] HybridClock initialized: now 1786086106214295 us; error 35 us; skew 500 ppm
I20260807 07:01:46.214675 26405 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:01:46 dist-test-slave-ncp9 krb5kdc[26385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086106, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:01:46.217700 26405 init.cc:377] Logged in from keytab as kudu/127.21.83.62@KRBTEST.COM (short username kudu)
I20260807 07:01:46.219053 26405 webserver.cc:460] Webserver started at http://127.21.83.62:35473/ using document root <none> and password file <none>
I20260807 07:01:46.219686 26405 fs_manager.cc:359] Metadata directory not provided
I20260807 07:01:46.219787 26405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:01:46.220082 26405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:01:46.223011 26405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "0fe035e65477498eb57398ccb3804b56"
format_stamp: "Formatted at 2026-08-07 07:01:46 on dist-test-slave-ncp9"
server_key: "1fe7ff5556782ac5a0a0951c86ba64e9"
server_key_iv: "aabd0a70c24d7a5df754e259af0e1ff3"
server_key_version: "encryptionkey@0"
I20260807 07:01:46.223767 26405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "0fe035e65477498eb57398ccb3804b56"
format_stamp: "Formatted at 2026-08-07 07:01:46 on dist-test-slave-ncp9"
server_key: "1fe7ff5556782ac5a0a0951c86ba64e9"
server_key_iv: "aabd0a70c24d7a5df754e259af0e1ff3"
server_key_version: "encryptionkey@0"
I20260807 07:01:46.229029 26405 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260807 07:01:46.232681 26426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:46.234814 26405 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260807 07:01:46.235108 26405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "0fe035e65477498eb57398ccb3804b56"
format_stamp: "Formatted at 2026-08-07 07:01:46 on dist-test-slave-ncp9"
server_key: "1fe7ff5556782ac5a0a0951c86ba64e9"
server_key_iv: "aabd0a70c24d7a5df754e259af0e1ff3"
server_key_version: "encryptionkey@0"
I20260807 07:01:46.235528 26405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:01:46.261255 26405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:01:46.265595 26405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:01:46.265878 26405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:01:46.284046 26405 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.62:43863
I20260807 07:01:46.284089 26501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.62:43863 every 8 connection(s)
I20260807 07:01:46.285425 26405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:01:46.289084 26502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:01:46.291767 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 26405
I20260807 07:01:46.291999 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:01:46.292783 21836 external_mini_cluster.cc:1428] Setting key 35cdd57f7c5200ef8a8abf36ac904ec3
I20260807 07:01:46.301069 26502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fe035e65477498eb57398ccb3804b56: Bootstrap starting.
I20260807 07:01:46.304693 26502 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fe035e65477498eb57398ccb3804b56: Neither blocks nor log segments found. Creating new log.
I20260807 07:01:46.305992 26502 log.cc:824] T 00000000000000000000000000000000 P 0fe035e65477498eb57398ccb3804b56: Log is configured to *not* fsync() on all Append() calls
Aug 07 07:01:46 dist-test-slave-ncp9 krb5kdc[26385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:01:46.308828 26502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fe035e65477498eb57398ccb3804b56: No bootstrap required, opened a new log
I20260807 07:01:46.316757 26505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:46.294672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38254 (local address 127.21.83.62:43863)
0807 07:01:46.295422 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:46.295433 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:46.295479 (+ 46us) server_negotiation.cc:407] Connection header received
0807 07:01:46.297165 (+ 1686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:46.297186 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:46.297577 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:46.297880 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:46.299641 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:46.301062 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:46.302785 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:46.303453 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:46.303584 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:46.308269 (+ 4685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:46.308276 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:46.308290 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:46.308325 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:46.312097 (+ 3772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:46.312812 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:46.312820 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:46.312828 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:46.313069 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:46.313501 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:46.313506 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:46.313509 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:46.314271 (+ 762us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:46.314576 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:46.314971 (+ 395us) server_negotiation.cc:299] Negotiation successful
0807 07:01:46.315426 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":347,"threads_started":1}
I20260807 07:01:46.318676 26502 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fe035e65477498eb57398ccb3804b56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fe035e65477498eb57398ccb3804b56" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 43863 } }
I20260807 07:01:46.319005 26502 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0fe035e65477498eb57398ccb3804b56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:01:46.319106 26502 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fe035e65477498eb57398ccb3804b56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe035e65477498eb57398ccb3804b56, State: Initialized, Role: FOLLOWER
I20260807 07:01:46.319718 26502 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fe035e65477498eb57398ccb3804b56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fe035e65477498eb57398ccb3804b56" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 43863 } }
I20260807 07:01:46.319942 26502 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0fe035e65477498eb57398ccb3804b56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:01:46.320075 26502 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fe035e65477498eb57398ccb3804b56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:01:46.320232 26502 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fe035e65477498eb57398ccb3804b56 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:01:46.320557 21836 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 07:01:46.323336 26502 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fe035e65477498eb57398ccb3804b56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fe035e65477498eb57398ccb3804b56" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 43863 } }
I20260807 07:01:46.323891 26502 leader_election.cc:302] T 00000000000000000000000000000000 P 0fe035e65477498eb57398ccb3804b56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0fe035e65477498eb57398ccb3804b56; no voters:
I20260807 07:01:46.324520 26502 leader_election.cc:288] T 00000000000000000000000000000000 P 0fe035e65477498eb57398ccb3804b56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 07:01:46.324640 26507 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fe035e65477498eb57398ccb3804b56 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:01:46.325796 26507 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0fe035e65477498eb57398ccb3804b56 [term 1 LEADER]: Becoming Leader. State: Replica: 0fe035e65477498eb57398ccb3804b56, State: Running, Role: LEADER
I20260807 07:01:46.326292 26507 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fe035e65477498eb57398ccb3804b56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fe035e65477498eb57398ccb3804b56" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 43863 } }
I20260807 07:01:46.326649 26502 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fe035e65477498eb57398ccb3804b56 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:01:46.328886 26510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fe035e65477498eb57398ccb3804b56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0fe035e65477498eb57398ccb3804b56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fe035e65477498eb57398ccb3804b56" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 43863 } } }
I20260807 07:01:46.328860 26511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fe035e65477498eb57398ccb3804b56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fe035e65477498eb57398ccb3804b56. Latest consensus state: current_term: 1 leader_uuid: "0fe035e65477498eb57398ccb3804b56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fe035e65477498eb57398ccb3804b56" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 43863 } } }
I20260807 07:01:46.329376 26510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fe035e65477498eb57398ccb3804b56 [sys.catalog]: This master's current role is: LEADER
I20260807 07:01:46.329447 26511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fe035e65477498eb57398ccb3804b56 [sys.catalog]: This master's current role is: LEADER
I20260807 07:01:46.343210 26520 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:01:46.348609 26520 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:01:46.368561 26520 catalog_manager.cc:1347] Generated new cluster ID: b3207a6583e042919b412fa195ba36d4
I20260807 07:01:46.368712 26520 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:01:46.378052 26520 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:01:46.380173 26520 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:01:46.385589 26505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:46.376960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38264 (local address 127.21.83.62:43863)
0807 07:01:46.377172 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:46.377178 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:46.377784 (+ 606us) server_negotiation.cc:407] Connection header received
0807 07:01:46.378070 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:46.378076 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:46.378154 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:46.378253 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:46.379500 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:46.380370 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:46.382089 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:46.382564 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:46.382808 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:46.383968 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:46.383974 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:46.383977 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:01:46.384639 (+ 662us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:46.384714 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:01:46.384745 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:46.384847 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:46.385063 (+ 216us) server_negotiation.cc:299] Negotiation successful
0807 07:01:46.385224 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260807 07:01:46.393603 26520 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0fe035e65477498eb57398ccb3804b56: Generated new TSK 0
I20260807 07:01:46.394824 26520 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 07:01:46.409701 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 26405
I20260807 07:01:46.411094 26495 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:01:46.502597 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 26405
2026-08-07T07:01:46Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2073 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 07:01:46 dist-test-slave-ncp9 krb5kdc[26550](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 07:01:46 dist-test-slave-ncp9 krb5kdc[26550](info): set up 2 sockets
Aug 07 07:01:46 dist-test-slave-ncp9 krb5kdc[26550](info): commencing operation
krb5kdc: starting...
W20260807 07:01:47.674981 21836 mini_kdc.cc:121] Time spent starting KDC: real 1.132s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Aug 07 07:01:47 dist-test-slave-ncp9 krb5kdc[26550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T07:01:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:01:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
I20260807 07:01:47.910559 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:45189
--webserver_interface=127.21.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:44223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:45189
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:01:48.231655 26571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:01:48.231927 26571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:01:48.232070 26571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:01:48.238930 26571 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:01:48.239034 26571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:01:48.239122 26571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:01:48.239202 26571 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:01:48.239262 26571 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:01:48.248389 26571 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:44223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:45189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:45189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:01:48.249529 26571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:01:48.250691 26571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:01:48.261410 26581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:48.261474 26578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:48.261448 26579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:01:48.262226 26571 server_base.cc:1034] running on GCE node
I20260807 07:01:48.262861 26571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:01:48.264101 26571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:01:48.265295 26571 hybrid_clock.cc:648] HybridClock initialized: now 1786086108265246 us; error 50 us; skew 500 ppm
I20260807 07:01:48.265617 26571 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:01:48 dist-test-slave-ncp9 krb5kdc[26550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086108, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:01:48.268576 26571 init.cc:377] Logged in from keytab as kudu/127.21.83.62@KRBTEST.COM (short username kudu)
I20260807 07:01:48.270020 26571 webserver.cc:460] Webserver started at http://127.21.83.62:45753/ using document root <none> and password file <none>
I20260807 07:01:48.270646 26571 fs_manager.cc:359] Metadata directory not provided
I20260807 07:01:48.270767 26571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:01:48.271049 26571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:01:48.273908 26571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "819300d89eb24ede961e7c78172be551"
format_stamp: "Formatted at 2026-08-07 07:01:48 on dist-test-slave-ncp9"
server_key: "726cbf5b5c3da4dd6d0c4fcb6090d867"
server_key_iv: "a9fef1e2a473140cb9b48e754f7c1d41"
server_key_version: "encryptionkey@0"
I20260807 07:01:48.274622 26571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "819300d89eb24ede961e7c78172be551"
format_stamp: "Formatted at 2026-08-07 07:01:48 on dist-test-slave-ncp9"
server_key: "726cbf5b5c3da4dd6d0c4fcb6090d867"
server_key_iv: "a9fef1e2a473140cb9b48e754f7c1d41"
server_key_version: "encryptionkey@0"
I20260807 07:01:48.279407 26571 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 07:01:48.282557 26590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:48.283747 26571 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260807 07:01:48.283900 26571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "819300d89eb24ede961e7c78172be551"
format_stamp: "Formatted at 2026-08-07 07:01:48 on dist-test-slave-ncp9"
server_key: "726cbf5b5c3da4dd6d0c4fcb6090d867"
server_key_iv: "a9fef1e2a473140cb9b48e754f7c1d41"
server_key_version: "encryptionkey@0"
I20260807 07:01:48.284075 26571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:01:48.295359 26571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:01:48.298827 26571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:01:48.299086 26571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:01:48.315945 26571 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.62:45189
I20260807 07:01:48.315989 26674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.62:45189 every 8 connection(s)
I20260807 07:01:48.317258 26571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:01:48.320281 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 26571
I20260807 07:01:48.320501 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:01:48.320842 26675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:01:48.321326 21836 external_mini_cluster.cc:1428] Setting key 5846957176178ef7472665e14abaf24d
I20260807 07:01:48.334342 26675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 819300d89eb24ede961e7c78172be551: Bootstrap starting.
Aug 07 07:01:48 dist-test-slave-ncp9 krb5kdc[26550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:01:48.337498 26675 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 819300d89eb24ede961e7c78172be551: Neither blocks nor log segments found. Creating new log.
I20260807 07:01:48.338464 26675 log.cc:824] T 00000000000000000000000000000000 P 819300d89eb24ede961e7c78172be551: Log is configured to *not* fsync() on all Append() calls
I20260807 07:01:48.341485 26675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 819300d89eb24ede961e7c78172be551: No bootstrap required, opened a new log
I20260807 07:01:48.343357 26679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:48.323283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56902 (local address 127.21.83.62:45189)
0807 07:01:48.324039 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:48.324049 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:48.324098 (+ 49us) server_negotiation.cc:407] Connection header received
0807 07:01:48.325667 (+ 1569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:48.325687 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:48.326027 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:48.326340 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:48.327954 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:48.329210 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:48.330846 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:48.331512 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:48.331652 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:48.336209 (+ 4557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:48.336217 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:48.336232 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:48.336270 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:48.339326 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:48.340044 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:48.340050 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:48.340055 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:48.340229 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:48.340636 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:48.340640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:48.340642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:48.341218 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:48.341489 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:48.341903 (+ 414us) server_negotiation.cc:299] Negotiation successful
0807 07:01:48.342358 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":352,"threads_started":1}
I20260807 07:01:48.349426 26675 raft_consensus.cc:348] T 00000000000000000000000000000000 P 819300d89eb24ede961e7c78172be551 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "819300d89eb24ede961e7c78172be551" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45189 } }
I20260807 07:01:48.349861 26675 raft_consensus.cc:374] T 00000000000000000000000000000000 P 819300d89eb24ede961e7c78172be551 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:01:48.349954 26675 raft_consensus.cc:729] T 00000000000000000000000000000000 P 819300d89eb24ede961e7c78172be551 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819300d89eb24ede961e7c78172be551, State: Initialized, Role: FOLLOWER
I20260807 07:01:48.350528 26675 consensus_queue.cc:260] T 00000000000000000000000000000000 P 819300d89eb24ede961e7c78172be551 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "819300d89eb24ede961e7c78172be551" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45189 } }
I20260807 07:01:48.350764 26675 raft_consensus.cc:388] T 00000000000000000000000000000000 P 819300d89eb24ede961e7c78172be551 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:01:48.350893 26675 raft_consensus.cc:482] T 00000000000000000000000000000000 P 819300d89eb24ede961e7c78172be551 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:01:48.351050 26675 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 819300d89eb24ede961e7c78172be551 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:01:48.353492 26675 raft_consensus.cc:504] T 00000000000000000000000000000000 P 819300d89eb24ede961e7c78172be551 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "819300d89eb24ede961e7c78172be551" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45189 } }
I20260807 07:01:48.354182 26675 leader_election.cc:302] T 00000000000000000000000000000000 P 819300d89eb24ede961e7c78172be551 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 819300d89eb24ede961e7c78172be551; no voters:
I20260807 07:01:48.354811 26675 leader_election.cc:288] T 00000000000000000000000000000000 P 819300d89eb24ede961e7c78172be551 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 07:01:48.354940 26681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 819300d89eb24ede961e7c78172be551 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:01:48.355603 26681 raft_consensus.cc:686] T 00000000000000000000000000000000 P 819300d89eb24ede961e7c78172be551 [term 1 LEADER]: Becoming Leader. State: Replica: 819300d89eb24ede961e7c78172be551, State: Running, Role: LEADER
I20260807 07:01:48.356146 26681 consensus_queue.cc:237] T 00000000000000000000000000000000 P 819300d89eb24ede961e7c78172be551 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "819300d89eb24ede961e7c78172be551" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45189 } }
I20260807 07:01:48.356993 26675 sys_catalog.cc:564] T 00000000000000000000000000000000 P 819300d89eb24ede961e7c78172be551 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:01:48.359088 26682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 819300d89eb24ede961e7c78172be551 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "819300d89eb24ede961e7c78172be551" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "819300d89eb24ede961e7c78172be551" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45189 } } }
I20260807 07:01:48.359058 26683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 819300d89eb24ede961e7c78172be551 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 819300d89eb24ede961e7c78172be551. Latest consensus state: current_term: 1 leader_uuid: "819300d89eb24ede961e7c78172be551" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "819300d89eb24ede961e7c78172be551" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45189 } } }
I20260807 07:01:48.359611 26682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 819300d89eb24ede961e7c78172be551 [sys.catalog]: This master's current role is: LEADER
I20260807 07:01:48.359611 26683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 819300d89eb24ede961e7c78172be551 [sys.catalog]: This master's current role is: LEADER
I20260807 07:01:48.362087 26686 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:01:48.367339 26686 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:01:48.375963 26686 catalog_manager.cc:1347] Generated new cluster ID: ced65e6724dc45749aa59214be3f90fe
I20260807 07:01:48.376083 26686 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:01:48.393715 26686 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:01:48.395025 26686 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:01:48.405800 26686 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 819300d89eb24ede961e7c78172be551: Generated new TSK 0
I20260807 07:01:48.406665 26686 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.83.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab.
Entry for principal kudu/127.21.83.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab.
I20260807 07:01:48.474000 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.1:0
--local_ip_for_outbound_sockets=127.21.83.1
--webserver_interface=127.21.83.1
--webserver_port=0
--tserver_master_addrs=127.21.83.62:45189
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab
--principal=kudu/127.21.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:44223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:01:48.825161 26708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:01:48.825414 26708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:01:48.825506 26708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:01:48.832485 26708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:01:48.832581 26708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:01:48.832712 26708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.1
I20260807 07:01:48.842396 26708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:44223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab
--principal=kudu/127.21.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.83.1
--webserver_port=0
--tserver_master_addrs=127.21.83.62:45189
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.1
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:01:48.843566 26708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:01:48.844801 26708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:01:48.856793 26714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:48.857160 26715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:48.857169 26719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:01:48.857895 26708 server_base.cc:1034] running on GCE node
I20260807 07:01:48.858491 26708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:01:48.859314 26708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:01:48.860519 26708 hybrid_clock.cc:648] HybridClock initialized: now 1786086108860474 us; error 36 us; skew 500 ppm
I20260807 07:01:48.860803 26708 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:01:48 dist-test-slave-ncp9 krb5kdc[26550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086108, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:01:48.863914 26708 init.cc:377] Logged in from keytab as kudu/127.21.83.1@KRBTEST.COM (short username kudu)
I20260807 07:01:48.865267 26708 webserver.cc:460] Webserver started at http://127.21.83.1:35647/ using document root <none> and password file <none>
I20260807 07:01:48.865856 26708 fs_manager.cc:359] Metadata directory not provided
I20260807 07:01:48.865953 26708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:01:48.866205 26708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:01:48.868916 26708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-0/data/instance:
uuid: "52bbc32015684b828155503a6b1b37ba"
format_stamp: "Formatted at 2026-08-07 07:01:48 on dist-test-slave-ncp9"
server_key: "26e1d33a9d0a6e2bed02e109c29f8e00"
server_key_iv: "ea73a6c572781f14dbba3405402d36b9"
server_key_version: "encryptionkey@0"
I20260807 07:01:48.869652 26708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-0/wal/instance:
uuid: "52bbc32015684b828155503a6b1b37ba"
format_stamp: "Formatted at 2026-08-07 07:01:48 on dist-test-slave-ncp9"
server_key: "26e1d33a9d0a6e2bed02e109c29f8e00"
server_key_iv: "ea73a6c572781f14dbba3405402d36b9"
server_key_version: "encryptionkey@0"
I20260807 07:01:48.874320 26708 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 07:01:48.877409 26730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:48.878603 26708 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 07:01:48.878726 26708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-0/wal
uuid: "52bbc32015684b828155503a6b1b37ba"
format_stamp: "Formatted at 2026-08-07 07:01:48 on dist-test-slave-ncp9"
server_key: "26e1d33a9d0a6e2bed02e109c29f8e00"
server_key_iv: "ea73a6c572781f14dbba3405402d36b9"
server_key_version: "encryptionkey@0"
I20260807 07:01:48.878875 26708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:01:48.898344 26708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:01:48.902411 26708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:01:48.902712 26708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:01:48.903748 26708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:01:48.905400 26708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:01:48.905507 26708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:48.905629 26708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:01:48.905686 26708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:48.935961 26708 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.1:46377
I20260807 07:01:48.936004 26898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.1:46377 every 8 connection(s)
I20260807 07:01:48.937290 26708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
I20260807 07:01:48.939241 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 26708
I20260807 07:01:48.939453 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-0/wal/instance
I20260807 07:01:48.940196 21836 external_mini_cluster.cc:1428] Setting key 0ccbf910b7204401c728cb23e8b5a42a
Aug 07 07:01:48 dist-test-slave-ncp9 krb5kdc[26550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086108, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.1@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:01:48.960367 26903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:48.940078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:35111 (local address 127.21.83.62:45189)
0807 07:01:48.940602 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:48.940609 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:48.942265 (+ 1656us) server_negotiation.cc:407] Connection header received
0807 07:01:48.944316 (+ 2051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:48.944322 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:48.944412 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:48.944511 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:48.946250 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:48.947024 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:48.948741 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:48.949221 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:48.949347 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:48.953523 (+ 4176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:48.953532 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:48.953538 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:48.953588 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:48.956730 (+ 3142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:48.957752 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:48.957758 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:48.957761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:48.957924 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:48.958420 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:48.958425 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:48.958428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:48.958814 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:48.958961 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:48.959773 (+ 812us) server_negotiation.cc:299] Negotiation successful
0807 07:01:48.960014 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":253,"threads_started":1}
I20260807 07:01:48.961292 26904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:48.940656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:45189 (local address 127.21.83.1:35111)
0807 07:01:48.941939 (+ 1283us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:48.942139 (+ 200us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:48.944013 (+ 1874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:48.944801 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:48.944811 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:48.945424 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:01:48.946058 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:48.946078 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:48.947240 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:48.947245 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:48.948566 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:48.948580 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:48.949487 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:48.949494 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:48.949904 (+ 410us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:48.950665 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:01:48.950696 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:01:48.953222 (+ 2526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:01:48.956989 (+ 3767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:48.956998 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:48.957014 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:48.957593 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:48.958079 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:48.958083 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:48.958086 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:48.958293 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:48.958977 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:01:48.958985 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:01:48.959352 (+ 367us) client_negotiation.cc:769] Sending connection context
0807 07:01:48.959618 (+ 266us) client_negotiation.cc:240] Negotiation successful
0807 07:01:48.960056 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":886,"thread_start_us":377,"threads_started":1}
I20260807 07:01:48.963099 26900 heartbeater.cc:347] Connected to a master server at 127.21.83.62:45189
I20260807 07:01:48.963487 26900 heartbeater.cc:464] Registering TS with master...
I20260807 07:01:48.964677 26900 heartbeater.cc:511] Master 127.21.83.62:45189 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.21.83.2@KRBTEST.COM; defaulting to no policy
I20260807 07:01:48.967235 26621 ts_manager.cc:194] Registered new tserver with Master: 52bbc32015684b828155503a6b1b37ba (127.21.83.1:46377)
I20260807 07:01:48.968801 26621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.1@KRBTEST.COM'} at 127.21.83.1:35111
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab.
Entry for principal kudu/127.21.83.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab.
I20260807 07:01:48.996011 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.2:0
--local_ip_for_outbound_sockets=127.21.83.2
--webserver_interface=127.21.83.2
--webserver_port=0
--tserver_master_addrs=127.21.83.62:45189
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab
--principal=kudu/127.21.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:44223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:01:49.335608 26909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:01:49.335856 26909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:01:49.335988 26909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:01:49.343240 26909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:01:49.343346 26909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:01:49.343503 26909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.2
I20260807 07:01:49.352797 26909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:44223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab
--principal=kudu/127.21.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.83.2
--webserver_port=0
--tserver_master_addrs=127.21.83.62:45189
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.2
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:01:49.353921 26909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:01:49.355151 26909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:01:49.372082 26917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:01:49.372466 26909 server_base.cc:1034] running on GCE node
W20260807 07:01:49.372535 26916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:49.372121 26919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:01:49.373433 26909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:01:49.374276 26909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:01:49.375473 26909 hybrid_clock.cc:648] HybridClock initialized: now 1786086109375420 us; error 40 us; skew 500 ppm
I20260807 07:01:49.375767 26909 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:01:49 dist-test-slave-ncp9 krb5kdc[26550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086109, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:01:49.378994 26909 init.cc:377] Logged in from keytab as kudu/127.21.83.2@KRBTEST.COM (short username kudu)
I20260807 07:01:49.380483 26909 webserver.cc:460] Webserver started at http://127.21.83.2:37841/ using document root <none> and password file <none>
I20260807 07:01:49.381270 26909 fs_manager.cc:359] Metadata directory not provided
I20260807 07:01:49.381423 26909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:01:49.381875 26909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:01:49.385264 26909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-1/data/instance:
uuid: "aef821017f61446a9fdd68b6cfef845d"
format_stamp: "Formatted at 2026-08-07 07:01:49 on dist-test-slave-ncp9"
server_key: "68411070add53142d17d41489493e15b"
server_key_iv: "f39031e25acdffc366af8068ea347127"
server_key_version: "encryptionkey@0"
I20260807 07:01:49.386165 26909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-1/wal/instance:
uuid: "aef821017f61446a9fdd68b6cfef845d"
format_stamp: "Formatted at 2026-08-07 07:01:49 on dist-test-slave-ncp9"
server_key: "68411070add53142d17d41489493e15b"
server_key_iv: "f39031e25acdffc366af8068ea347127"
server_key_version: "encryptionkey@0"
I20260807 07:01:49.392254 26909 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260807 07:01:49.396229 26929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:49.397909 26909 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 07:01:49.398100 26909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-1/wal
uuid: "aef821017f61446a9fdd68b6cfef845d"
format_stamp: "Formatted at 2026-08-07 07:01:49 on dist-test-slave-ncp9"
server_key: "68411070add53142d17d41489493e15b"
server_key_iv: "f39031e25acdffc366af8068ea347127"
server_key_version: "encryptionkey@0"
I20260807 07:01:49.398313 26909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:01:49.443748 26909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:01:49.449398 26909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:01:49.449767 26909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:01:49.450969 26909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:01:49.452324 26909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:01:49.452404 26909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:49.452524 26909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:01:49.452580 26909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:49.484146 26909 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.2:40227
I20260807 07:01:49.484230 27095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.2:40227 every 8 connection(s)
I20260807 07:01:49.485498 26909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
I20260807 07:01:49.489763 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 26909
I20260807 07:01:49.489951 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-1/wal/instance
I20260807 07:01:49.490513 21836 external_mini_cluster.cc:1428] Setting key 426b3a5a87ff1b68fb576b62beb9cb71
Aug 07 07:01:49 dist-test-slave-ncp9 krb5kdc[26550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086109, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.2@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:01:49.507122 27098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:49.488071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:54001 (local address 127.21.83.62:45189)
0807 07:01:49.488598 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:49.488604 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:49.489593 (+ 989us) server_negotiation.cc:407] Connection header received
0807 07:01:49.491936 (+ 2343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:49.491942 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:49.492018 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:49.492145 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:49.494139 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:49.494873 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:49.496974 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:49.497401 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:49.497498 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:49.501498 (+ 4000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:49.501503 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:49.501507 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:49.501540 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:49.504148 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:49.504838 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:49.504843 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:49.504845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:49.505006 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:49.505429 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:49.505432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:49.505434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:49.505701 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:49.505833 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:49.506505 (+ 672us) server_negotiation.cc:299] Negotiation successful
0807 07:01:49.506739 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":265,"threads_started":1}
I20260807 07:01:49.508033 27100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:49.488640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:45189 (local address 127.21.83.2:54001)
0807 07:01:49.489395 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:49.489461 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:49.491677 (+ 2216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:49.492373 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:49.492386 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:49.493125 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:01:49.493913 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:49.493937 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:49.495030 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:49.495036 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:49.496785 (+ 1749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:49.496826 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:49.497619 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:49.497626 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:49.497896 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:49.498579 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:01:49.498608 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:01:49.501269 (+ 2661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:01:49.504278 (+ 3009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:49.504287 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:49.504304 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:49.504716 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:49.505117 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:49.505122 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:49.505126 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:49.505313 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:49.505825 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:01:49.505832 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:01:49.506159 (+ 327us) client_negotiation.cc:769] Sending connection context
0807 07:01:49.506393 (+ 234us) client_negotiation.cc:240] Negotiation successful
0807 07:01:49.506898 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":300,"threads_started":1}
I20260807 07:01:49.509601 27096 heartbeater.cc:347] Connected to a master server at 127.21.83.62:45189
I20260807 07:01:49.509873 27096 heartbeater.cc:464] Registering TS with master...
I20260807 07:01:49.510564 27096 heartbeater.cc:511] Master 127.21.83.62:45189 requested a full tablet report, sending...
I20260807 07:01:49.512095 26621 ts_manager.cc:194] Registered new tserver with Master: aef821017f61446a9fdd68b6cfef845d (127.21.83.2:40227)
WARNING: no policy specified for kudu/127.21.83.3@KRBTEST.COM; defaulting to no policy
I20260807 07:01:49.513285 26621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.2@KRBTEST.COM'} at 127.21.83.2:54001
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab.
Entry for principal kudu/127.21.83.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab.
I20260807 07:01:49.539885 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.3:0
--local_ip_for_outbound_sockets=127.21.83.3
--webserver_interface=127.21.83.3
--webserver_port=0
--tserver_master_addrs=127.21.83.62:45189
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab
--principal=kudu/127.21.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:44223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:01:49.872967 27105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:01:49.873243 27105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:01:49.873345 27105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:01:49.880358 27105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:01:49.880451 27105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:01:49.880574 27105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.3
I20260807 07:01:49.890232 27105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:44223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab
--principal=kudu/127.21.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.83.3
--webserver_port=0
--tserver_master_addrs=127.21.83.62:45189
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.3
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:01:49.891388 27105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:01:49.892634 27105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:01:49.905300 27113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:49.905236 27116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:49.905272 27114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:01:49.906356 27105 server_base.cc:1034] running on GCE node
I20260807 07:01:49.906960 27105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:01:49.907954 27105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:01:49.909162 27105 hybrid_clock.cc:648] HybridClock initialized: now 1786086109909113 us; error 38 us; skew 500 ppm
I20260807 07:01:49.909458 27105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:01:49 dist-test-slave-ncp9 krb5kdc[26550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086109, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:01:49.912568 27105 init.cc:377] Logged in from keytab as kudu/127.21.83.3@KRBTEST.COM (short username kudu)
I20260807 07:01:49.914038 27105 webserver.cc:460] Webserver started at http://127.21.83.3:44577/ using document root <none> and password file <none>
I20260807 07:01:49.914700 27105 fs_manager.cc:359] Metadata directory not provided
I20260807 07:01:49.914803 27105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:01:49.915069 27105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:01:49.918063 27105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-2/data/instance:
uuid: "8f53af25f5cc45e99724658a66da8aa3"
format_stamp: "Formatted at 2026-08-07 07:01:49 on dist-test-slave-ncp9"
server_key: "5e0066e8afbe5f24d691e907d5aaa3fd"
server_key_iv: "83f10c8307a0ef9022d1beb5c4408ca6"
server_key_version: "encryptionkey@0"
I20260807 07:01:49.918810 27105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-2/wal/instance:
uuid: "8f53af25f5cc45e99724658a66da8aa3"
format_stamp: "Formatted at 2026-08-07 07:01:49 on dist-test-slave-ncp9"
server_key: "5e0066e8afbe5f24d691e907d5aaa3fd"
server_key_iv: "83f10c8307a0ef9022d1beb5c4408ca6"
server_key_version: "encryptionkey@0"
I20260807 07:01:49.924046 27105 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260807 07:01:49.927489 27128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:49.928819 27105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260807 07:01:49.929006 27105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-2/wal
uuid: "8f53af25f5cc45e99724658a66da8aa3"
format_stamp: "Formatted at 2026-08-07 07:01:49 on dist-test-slave-ncp9"
server_key: "5e0066e8afbe5f24d691e907d5aaa3fd"
server_key_iv: "83f10c8307a0ef9022d1beb5c4408ca6"
server_key_version: "encryptionkey@0"
I20260807 07:01:49.929194 27105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:01:49.952193 27105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:01:49.956804 27105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:01:49.957165 27105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:01:49.958304 27105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:01:49.959801 27105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:01:49.959892 27105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:49.960026 27105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:01:49.960081 27105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:49.972131 26900 heartbeater.cc:503] Master 127.21.83.62:45189 was elected leader, sending a full tablet report...
I20260807 07:01:49.992290 27105 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.3:32823
I20260807 07:01:49.992323 27298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.3:32823 every 8 connection(s)
I20260807 07:01:49.993698 27105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
I20260807 07:01:50.004204 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 27105
I20260807 07:01:50.004410 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786086078080656-21836-0/minicluster-data/ts-2/wal/instance
I20260807 07:01:50.004993 21836 external_mini_cluster.cc:1428] Setting key 742a4cc28594750efcbbc32dff8089d7
Aug 07 07:01:50 dist-test-slave-ncp9 krb5kdc[26550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086109, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.3@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:01:50.013891 27098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:49.996274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:34351 (local address 127.21.83.62:45189)
0807 07:01:49.996473 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:49.996477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:49.997683 (+ 1206us) server_negotiation.cc:407] Connection header received
0807 07:01:49.999525 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:49.999530 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:49.999632 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:49.999727 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:50.001307 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:50.002073 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:50.003764 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:50.004350 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:50.004499 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:50.008223 (+ 3724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:50.008228 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:50.008232 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:50.008266 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:50.010898 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:50.011628 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:50.011632 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:50.011634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:50.011758 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:50.012207 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:50.012210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:50.012211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:50.012486 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:50.012607 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:50.013458 (+ 851us) server_negotiation.cc:299] Negotiation successful
0807 07:01:50.013655 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 07:01:50.014810 27302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:49.996729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:45189 (local address 127.21.83.3:34351)
0807 07:01:49.997514 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:49.997559 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:49.999255 (+ 1696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:49.999915 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:49.999922 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:50.000460 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:01:50.001106 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:50.001131 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:50.002246 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:50.002250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:50.003583 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:50.003600 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:50.004599 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:50.004605 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:50.004816 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:50.005501 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:01:50.005525 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:01:50.008026 (+ 2501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:01:50.011035 (+ 3009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:50.011043 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:50.011056 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:50.011506 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:50.011891 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:50.011895 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:50.011898 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:50.012115 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:50.012635 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:01:50.012642 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:01:50.013050 (+ 408us) client_negotiation.cc:769] Sending connection context
0807 07:01:50.013296 (+ 246us) client_negotiation.cc:240] Negotiation successful
0807 07:01:50.013719 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":327,"threads_started":1}
I20260807 07:01:50.016136 27299 heartbeater.cc:347] Connected to a master server at 127.21.83.62:45189
I20260807 07:01:50.016439 27299 heartbeater.cc:464] Registering TS with master...
I20260807 07:01:50.017225 27299 heartbeater.cc:511] Master 127.21.83.62:45189 requested a full tablet report, sending...
I20260807 07:01:50.018810 26621 ts_manager.cc:194] Registered new tserver with Master: 8f53af25f5cc45e99724658a66da8aa3 (127.21.83.3:32823)
I20260807 07:01:50.019759 26621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.3@KRBTEST.COM'} at 127.21.83.3:34351
I20260807 07:01:50.020527 21836 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 07:01:50 dist-test-slave-ncp9 krb5kdc[26550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086110, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 07 07:01:50 dist-test-slave-ncp9 krb5kdc[26550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086110, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:01:50.078924 27098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:50.064153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55604 (local address 127.21.83.62:45189)
0807 07:01:50.064416 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:50.064423 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:50.064658 (+ 235us) server_negotiation.cc:407] Connection header received
0807 07:01:50.064802 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:50.064806 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:50.064911 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:50.065015 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:50.066431 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:50.067218 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:50.068472 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:50.068957 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:50.069063 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:50.073539 (+ 4476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:50.073545 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:50.073550 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:50.073598 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:50.076295 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:50.077092 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:50.077096 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:50.077099 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:50.077245 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:50.077667 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:50.077672 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:50.077674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:50.078050 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:50.078277 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:50.078467 (+ 190us) server_negotiation.cc:299] Negotiation successful
0807 07:01:50.078662 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260807 07:01:50.079551 26621 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:55604
I20260807 07:01:50.081601 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 26708
Aug 07 07:01:50 dist-test-slave-ncp9 krb5kdc[26550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086110, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.21.83.1@KRBTEST.COM
I20260807 07:01:50.097554 27314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:50.082670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55572 (local address 127.21.83.1:46377)
0807 07:01:50.083373 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:50.083382 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:50.083408 (+ 26us) server_negotiation.cc:407] Connection header received
0807 07:01:50.083468 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:50.083475 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:50.083700 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:50.083823 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:50.085162 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:50.086244 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:50.087546 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:50.088212 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:50.088315 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:50.092402 (+ 4087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:50.092416 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:50.092431 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:50.092469 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:50.095162 (+ 2693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:50.095782 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:50.095788 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:50.095790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:50.095910 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:50.096186 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:50.096188 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:50.096190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:50.096513 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:50.096757 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:50.096980 (+ 223us) server_negotiation.cc:299] Negotiation successful
0807 07:01:50.097186 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":383,"threads_started":1}
W20260807 07:01:50.098810 26890 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:55572
W20260807 07:01:50.099943 21836 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 26708: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 07:01:50.100162 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 26708
I20260807 07:01:50.122066 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 26909
Aug 07 07:01:50 dist-test-slave-ncp9 krb5kdc[26550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086110, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.21.83.2@KRBTEST.COM
I20260807 07:01:50.137460 27316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:50.122938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58268 (local address 127.21.83.2:40227)
0807 07:01:50.123511 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:50.123519 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:50.123545 (+ 26us) server_negotiation.cc:407] Connection header received
0807 07:01:50.123606 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:50.123612 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:50.123834 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:50.123964 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:50.125187 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:50.126206 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:50.127400 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:50.127943 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:50.128040 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:50.132078 (+ 4038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:50.132091 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:50.132106 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:50.132144 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:50.134901 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:50.135520 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:50.135527 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:50.135530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:50.135659 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:50.135983 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:50.135986 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:50.135988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:50.136317 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:50.136545 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:50.136816 (+ 271us) server_negotiation.cc:299] Negotiation successful
0807 07:01:50.137064 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":286,"threads_started":1}
W20260807 07:01:50.138949 27089 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58268
W20260807 07:01:50.140151 21836 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 26909: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 07:01:50.140296 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 26909
I20260807 07:01:50.161319 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 27105
Aug 07 07:01:50 dist-test-slave-ncp9 krb5kdc[26550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086110, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.21.83.3@KRBTEST.COM
I20260807 07:01:50.177793 27317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:50.162455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47424 (local address 127.21.83.3:32823)
0807 07:01:50.163116 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:50.163124 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:50.163153 (+ 29us) server_negotiation.cc:407] Connection header received
0807 07:01:50.163215 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:50.163222 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:50.163489 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:50.163643 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:50.165101 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:50.166202 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:50.167489 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:50.168066 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:50.168181 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:50.172367 (+ 4186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:50.172379 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:50.172394 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:50.172431 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:50.175080 (+ 2649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:50.175700 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:50.175707 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:50.175710 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:50.175835 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:50.176192 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:50.176195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:50.176197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:50.176541 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:50.176834 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:50.177220 (+ 386us) server_negotiation.cc:299] Negotiation successful
0807 07:01:50.177489 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":335,"threads_started":1}
W20260807 07:01:50.179011 27291 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47424
W20260807 07:01:50.180017 21836 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 27105: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 07:01:50.180181 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 27105
I20260807 07:01:50.203930 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 26571
I20260807 07:01:50.205107 26669 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:01:50.307977 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 26571
2026-08-07T07:01:50Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3810 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 07:01:50 dist-test-slave-ncp9 krb5kdc[27322](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 07:01:50 dist-test-slave-ncp9 krb5kdc[27322](info): set up 2 sockets
Aug 07 07:01:50 dist-test-slave-ncp9 krb5kdc[27322](info): commencing operation
krb5kdc: starting...
W20260807 07:01:51.492777 21836 mini_kdc.cc:121] Time spent starting KDC: real 1.140s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Aug 07 07:01:51 dist-test-slave-ncp9 krb5kdc[27322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T07:01:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:01:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
I20260807 07:01:51.726089 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:32885
--webserver_interface=127.21.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:34019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:32885
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:01:52.063323 27341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:01:52.063634 27341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:01:52.063773 27341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:01:52.070989 27341 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:01:52.071115 27341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:01:52.071170 27341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:01:52.071259 27341 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:01:52.071324 27341 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:01:52.081696 27341 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:34019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:32885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:32885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:01:52.083153 27341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:01:52.084494 27341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:01:52.096140 27348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:52.096211 27347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:52.096668 27353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:01:52.096800 27341 server_base.cc:1034] running on GCE node
I20260807 07:01:52.097697 27341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:01:52.099154 27341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:01:52.100355 27341 hybrid_clock.cc:648] HybridClock initialized: now 1786086112100302 us; error 47 us; skew 500 ppm
I20260807 07:01:52.100719 27341 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:01:52 dist-test-slave-ncp9 krb5kdc[27322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086112, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:01:52.110095 27341 init.cc:377] Logged in from keytab as kudu/127.21.83.62@KRBTEST.COM (short username kudu)
I20260807 07:01:52.111886 27341 webserver.cc:460] Webserver started at http://127.21.83.62:38207/ using document root <none> and password file <none>
I20260807 07:01:52.112613 27341 fs_manager.cc:359] Metadata directory not provided
I20260807 07:01:52.112742 27341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:01:52.113106 27341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:01:52.116691 27341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "0fdde737c5cc44a99188bb59dec1e653"
format_stamp: "Formatted at 2026-08-07 07:01:52 on dist-test-slave-ncp9"
server_key: "f2859afc2357e0d3a587bd3165c9790c"
server_key_iv: "2e5219d2ea81ac61f0e36bb3c0767e26"
server_key_version: "encryptionkey@0"
I20260807 07:01:52.117597 27341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "0fdde737c5cc44a99188bb59dec1e653"
format_stamp: "Formatted at 2026-08-07 07:01:52 on dist-test-slave-ncp9"
server_key: "f2859afc2357e0d3a587bd3165c9790c"
server_key_iv: "2e5219d2ea81ac61f0e36bb3c0767e26"
server_key_version: "encryptionkey@0"
I20260807 07:01:52.123035 27341 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 07:01:52.126788 27367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:52.128193 27341 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 07:01:52.128379 27341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "0fdde737c5cc44a99188bb59dec1e653"
format_stamp: "Formatted at 2026-08-07 07:01:52 on dist-test-slave-ncp9"
server_key: "f2859afc2357e0d3a587bd3165c9790c"
server_key_iv: "2e5219d2ea81ac61f0e36bb3c0767e26"
server_key_version: "encryptionkey@0"
I20260807 07:01:52.128590 27341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:01:52.148223 27341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:01:52.152671 27341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:01:52.153008 27341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:01:52.172096 27341 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.62:32885
I20260807 07:01:52.172128 27445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.62:32885 every 8 connection(s)
I20260807 07:01:52.173614 27341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:01:52.177521 27446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:01:52.180505 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 27341
I20260807 07:01:52.180758 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:01:52.181599 21836 external_mini_cluster.cc:1428] Setting key d8afb0d6097dcaf98fad971b4fe35326
I20260807 07:01:52.191335 27446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fdde737c5cc44a99188bb59dec1e653: Bootstrap starting.
I20260807 07:01:52.195122 27446 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fdde737c5cc44a99188bb59dec1e653: Neither blocks nor log segments found. Creating new log.
I20260807 07:01:52.196465 27446 log.cc:824] T 00000000000000000000000000000000 P 0fdde737c5cc44a99188bb59dec1e653: Log is configured to *not* fsync() on all Append() calls
Aug 07 07:01:52 dist-test-slave-ncp9 krb5kdc[27322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:01:52.199757 27446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fdde737c5cc44a99188bb59dec1e653: No bootstrap required, opened a new log
I20260807 07:01:52.206151 27450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:52.183996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54594 (local address 127.21.83.62:32885)
0807 07:01:52.185042 (+ 1046us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:52.185058 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:52.185362 (+ 304us) server_negotiation.cc:407] Connection header received
0807 07:01:52.187236 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:52.187262 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:52.187706 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:52.188046 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:52.190000 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:52.191301 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:52.193101 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:52.193779 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:52.193911 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:52.198875 (+ 4964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:52.198883 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:52.198897 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:52.198938 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:52.202026 (+ 3088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:52.202873 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:52.202879 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:52.202885 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:52.203055 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:52.203455 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:52.203458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:52.203463 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:52.203996 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:52.204239 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:52.204682 (+ 443us) server_negotiation.cc:299] Negotiation successful
0807 07:01:52.205088 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":725,"thread_start_us":427,"threads_started":1}
I20260807 07:01:52.208676 27446 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fdde737c5cc44a99188bb59dec1e653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fdde737c5cc44a99188bb59dec1e653" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 32885 } }
I20260807 07:01:52.209115 27446 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0fdde737c5cc44a99188bb59dec1e653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:01:52.209237 27446 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fdde737c5cc44a99188bb59dec1e653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fdde737c5cc44a99188bb59dec1e653, State: Initialized, Role: FOLLOWER
I20260807 07:01:52.209748 27446 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fdde737c5cc44a99188bb59dec1e653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fdde737c5cc44a99188bb59dec1e653" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 32885 } }
I20260807 07:01:52.209966 27446 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0fdde737c5cc44a99188bb59dec1e653 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:01:52.210108 27446 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fdde737c5cc44a99188bb59dec1e653 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:01:52.210258 27446 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fdde737c5cc44a99188bb59dec1e653 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:01:52.212697 27446 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fdde737c5cc44a99188bb59dec1e653 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fdde737c5cc44a99188bb59dec1e653" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 32885 } }
I20260807 07:01:52.213234 27446 leader_election.cc:302] T 00000000000000000000000000000000 P 0fdde737c5cc44a99188bb59dec1e653 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0fdde737c5cc44a99188bb59dec1e653; no voters:
I20260807 07:01:52.213812 27446 leader_election.cc:288] T 00000000000000000000000000000000 P 0fdde737c5cc44a99188bb59dec1e653 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 07:01:52.214002 27454 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fdde737c5cc44a99188bb59dec1e653 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:01:52.215092 27454 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0fdde737c5cc44a99188bb59dec1e653 [term 1 LEADER]: Becoming Leader. State: Replica: 0fdde737c5cc44a99188bb59dec1e653, State: Running, Role: LEADER
I20260807 07:01:52.215583 27454 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fdde737c5cc44a99188bb59dec1e653 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fdde737c5cc44a99188bb59dec1e653" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 32885 } }
I20260807 07:01:52.215996 27446 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fdde737c5cc44a99188bb59dec1e653 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:01:52.218128 27457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fdde737c5cc44a99188bb59dec1e653 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fdde737c5cc44a99188bb59dec1e653. Latest consensus state: current_term: 1 leader_uuid: "0fdde737c5cc44a99188bb59dec1e653" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fdde737c5cc44a99188bb59dec1e653" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 32885 } } }
I20260807 07:01:52.218170 27456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fdde737c5cc44a99188bb59dec1e653 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0fdde737c5cc44a99188bb59dec1e653" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fdde737c5cc44a99188bb59dec1e653" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 32885 } } }
I20260807 07:01:52.218600 27456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fdde737c5cc44a99188bb59dec1e653 [sys.catalog]: This master's current role is: LEADER
I20260807 07:01:52.218605 27457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fdde737c5cc44a99188bb59dec1e653 [sys.catalog]: This master's current role is: LEADER
I20260807 07:01:52.224231 27463 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:01:52.228262 27463 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:01:52.237192 27463 catalog_manager.cc:1347] Generated new cluster ID: 38f6c9f0a887483183cd0bd629779bf2
I20260807 07:01:52.237380 27463 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:01:52.258112 27463 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:01:52.259446 27463 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:01:52.265817 27463 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0fdde737c5cc44a99188bb59dec1e653: Generated new TSK 0
I20260807 07:01:52.266603 27463 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.83.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab.
Entry for principal kudu/127.21.83.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab.
I20260807 07:01:52.324256 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.1:0
--local_ip_for_outbound_sockets=127.21.83.1
--webserver_interface=127.21.83.1
--webserver_port=0
--tserver_master_addrs=127.21.83.62:32885
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab
--principal=kudu/127.21.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:34019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:01:52.671921 27482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:01:52.672212 27482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:01:52.672353 27482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:01:52.679703 27482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:01:52.679864 27482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:01:52.680029 27482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.1
I20260807 07:01:52.690493 27482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:34019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab
--principal=kudu/127.21.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.83.1
--webserver_port=0
--tserver_master_addrs=127.21.83.62:32885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.1
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:01:52.691910 27482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:01:52.693420 27482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:01:52.707010 27491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:52.706964 27496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:52.707149 27493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:01:52.707971 27482 server_base.cc:1034] running on GCE node
I20260807 07:01:52.708622 27482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:01:52.709620 27482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:01:52.710839 27482 hybrid_clock.cc:648] HybridClock initialized: now 1786086112710786 us; error 44 us; skew 500 ppm
I20260807 07:01:52.711161 27482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:01:52 dist-test-slave-ncp9 krb5kdc[27322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086112, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:01:52.714596 27482 init.cc:377] Logged in from keytab as kudu/127.21.83.1@KRBTEST.COM (short username kudu)
I20260807 07:01:52.716140 27482 webserver.cc:460] Webserver started at http://127.21.83.1:37283/ using document root <none> and password file <none>
I20260807 07:01:52.716885 27482 fs_manager.cc:359] Metadata directory not provided
I20260807 07:01:52.717052 27482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:01:52.717368 27482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:01:52.720486 27482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/ts-0/data/instance:
uuid: "e2cbc96050d64cdf95f6d129c1fa58d3"
format_stamp: "Formatted at 2026-08-07 07:01:52 on dist-test-slave-ncp9"
server_key: "5abb2d7e7bdc9e7f7ae54b28ea592908"
server_key_iv: "baf5fc475db86e3e87f652aaf03ee9c6"
server_key_version: "encryptionkey@0"
I20260807 07:01:52.721331 27482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/ts-0/wal/instance:
uuid: "e2cbc96050d64cdf95f6d129c1fa58d3"
format_stamp: "Formatted at 2026-08-07 07:01:52 on dist-test-slave-ncp9"
server_key: "5abb2d7e7bdc9e7f7ae54b28ea592908"
server_key_iv: "baf5fc475db86e3e87f652aaf03ee9c6"
server_key_version: "encryptionkey@0"
I20260807 07:01:52.726395 27482 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260807 07:01:52.730093 27504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:52.731489 27482 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 07:01:52.731678 27482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/ts-0/wal
uuid: "e2cbc96050d64cdf95f6d129c1fa58d3"
format_stamp: "Formatted at 2026-08-07 07:01:52 on dist-test-slave-ncp9"
server_key: "5abb2d7e7bdc9e7f7ae54b28ea592908"
server_key_iv: "baf5fc475db86e3e87f652aaf03ee9c6"
server_key_version: "encryptionkey@0"
I20260807 07:01:52.731899 27482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:01:52.751804 27482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:01:52.755740 27482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:01:52.756016 27482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:01:52.757072 27482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:01:52.758517 27482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:01:52.758605 27482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:52.758738 27482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:01:52.758792 27482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:52.791342 27482 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.1:46699
I20260807 07:01:52.791389 27670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.1:46699 every 8 connection(s)
I20260807 07:01:52.792709 27482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
I20260807 07:01:52.800180 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 27482
I20260807 07:01:52.800393 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/ts-0/wal/instance
I20260807 07:01:52.801213 21836 external_mini_cluster.cc:1428] Setting key 7091075451f6b45550cf6102c0730322
Aug 07 07:01:52 dist-test-slave-ncp9 krb5kdc[27322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086112, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.1@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:01:52.813752 27674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:52.795372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:53813 (local address 127.21.83.62:32885)
0807 07:01:52.795822 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:52.795826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:52.796959 (+ 1133us) server_negotiation.cc:407] Connection header received
0807 07:01:52.798861 (+ 1902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:52.798866 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:52.798941 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:52.799045 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:52.800740 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:52.801819 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:52.803696 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:52.804262 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:52.804384 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:52.808073 (+ 3689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:52.808078 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:52.808082 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:52.808117 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:52.810507 (+ 2390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:52.811333 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:52.811336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:52.811339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:52.811450 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:52.811912 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:52.811917 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:52.811920 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:52.812279 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:52.812429 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:52.813170 (+ 741us) server_negotiation.cc:299] Negotiation successful
0807 07:01:52.813421 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":219,"threads_started":1}
I20260807 07:01:52.814627 27675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:52.795899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:32885 (local address 127.21.83.1:53813)
0807 07:01:52.796758 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:52.796806 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:52.798544 (+ 1738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:52.799265 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:52.799275 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:52.799872 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:01:52.800541 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:52.800564 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:52.802019 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:52.802024 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:52.803539 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:52.803553 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:52.804473 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:52.804478 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:52.804690 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:52.805396 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:01:52.805425 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:01:52.807836 (+ 2411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:01:52.810655 (+ 2819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:52.810664 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:52.810681 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:52.811235 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:52.811541 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:52.811545 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:52.811548 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:52.811805 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:52.812389 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:01:52.812395 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:01:52.812737 (+ 342us) client_negotiation.cc:769] Sending connection context
0807 07:01:52.813037 (+ 300us) client_negotiation.cc:240] Negotiation successful
0807 07:01:52.813471 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":362,"threads_started":1}
I20260807 07:01:52.816224 27671 heartbeater.cc:347] Connected to a master server at 127.21.83.62:32885
I20260807 07:01:52.816551 27671 heartbeater.cc:464] Registering TS with master...
I20260807 07:01:52.817561 27671 heartbeater.cc:511] Master 127.21.83.62:32885 requested a full tablet report, sending...
I20260807 07:01:52.819986 27393 ts_manager.cc:194] Registered new tserver with Master: e2cbc96050d64cdf95f6d129c1fa58d3 (127.21.83.1:46699)
I20260807 07:01:52.822007 27393 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.1@KRBTEST.COM'} at 127.21.83.1:53813
WARNING: no policy specified for kudu/127.21.83.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab.
Entry for principal kudu/127.21.83.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab.
I20260807 07:01:52.856364 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.2:0
--local_ip_for_outbound_sockets=127.21.83.2
--webserver_interface=127.21.83.2
--webserver_port=0
--tserver_master_addrs=127.21.83.62:32885
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab
--principal=kudu/127.21.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:34019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:01:53.207702 27680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:01:53.208056 27680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:01:53.208164 27680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:01:53.216655 27680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:01:53.216812 27680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:01:53.217003 27680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.2
I20260807 07:01:53.227950 27680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:34019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab
--principal=kudu/127.21.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.83.2
--webserver_port=0
--tserver_master_addrs=127.21.83.62:32885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.2
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:01:53.229339 27680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:01:53.230710 27680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:01:53.244779 27694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:53.244706 27691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:53.244801 27690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:01:53.245627 27680 server_base.cc:1034] running on GCE node
I20260807 07:01:53.246318 27680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:01:53.247394 27680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:01:53.248643 27680 hybrid_clock.cc:648] HybridClock initialized: now 1786086113248583 us; error 54 us; skew 500 ppm
I20260807 07:01:53.249040 27680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:01:53 dist-test-slave-ncp9 krb5kdc[27322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086113, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:01:53.252179 27680 init.cc:377] Logged in from keytab as kudu/127.21.83.2@KRBTEST.COM (short username kudu)
I20260807 07:01:53.253754 27680 webserver.cc:460] Webserver started at http://127.21.83.2:46345/ using document root <none> and password file <none>
I20260807 07:01:53.254449 27680 fs_manager.cc:359] Metadata directory not provided
I20260807 07:01:53.254575 27680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:01:53.254909 27680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:01:53.258463 27680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/ts-1/data/instance:
uuid: "02bf35445aad4acaa61f6fe4dc300dc7"
format_stamp: "Formatted at 2026-08-07 07:01:53 on dist-test-slave-ncp9"
server_key: "a9b4c51a0d774f11b9e40afe41b951be"
server_key_iv: "8cd7c9d79888ac1b37a384140371ddaa"
server_key_version: "encryptionkey@0"
I20260807 07:01:53.259443 27680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/ts-1/wal/instance:
uuid: "02bf35445aad4acaa61f6fe4dc300dc7"
format_stamp: "Formatted at 2026-08-07 07:01:53 on dist-test-slave-ncp9"
server_key: "a9b4c51a0d774f11b9e40afe41b951be"
server_key_iv: "8cd7c9d79888ac1b37a384140371ddaa"
server_key_version: "encryptionkey@0"
I20260807 07:01:53.265431 27680 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260807 07:01:53.269549 27704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:53.271026 27680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20260807 07:01:53.271211 27680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/ts-1/wal
uuid: "02bf35445aad4acaa61f6fe4dc300dc7"
format_stamp: "Formatted at 2026-08-07 07:01:53 on dist-test-slave-ncp9"
server_key: "a9b4c51a0d774f11b9e40afe41b951be"
server_key_iv: "8cd7c9d79888ac1b37a384140371ddaa"
server_key_version: "encryptionkey@0"
I20260807 07:01:53.271428 27680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:01:53.295826 27680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:01:53.300493 27680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:01:53.300810 27680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:01:53.301977 27680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:01:53.303470 27680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:01:53.303555 27680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:53.303689 27680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:01:53.303745 27680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:53.337001 27680 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.2:43903
I20260807 07:01:53.337050 27889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.2:43903 every 8 connection(s)
I20260807 07:01:53.338303 27680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
I20260807 07:01:53.342922 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 27680
I20260807 07:01:53.343143 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/ts-1/wal/instance
I20260807 07:01:53.343675 21836 external_mini_cluster.cc:1428] Setting key 839eef30275d653b93ce20d46b937b94
Aug 07 07:01:53 dist-test-slave-ncp9 krb5kdc[27322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086113, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.2@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:01:53.359704 27894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:53.340749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:38955 (local address 127.21.83.62:32885)
0807 07:01:53.341194 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:53.341198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:53.342114 (+ 916us) server_negotiation.cc:407] Connection header received
0807 07:01:53.344747 (+ 2633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:53.344754 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:53.344844 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:53.344981 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:53.346569 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:53.347589 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:53.349252 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:53.349805 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:53.349927 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:53.353356 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:53.353362 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:53.353365 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:53.353401 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:53.356372 (+ 2971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:53.357229 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:53.357233 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:53.357236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:53.357379 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:53.357844 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:53.357848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:53.357850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:53.358147 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:53.358284 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:53.359067 (+ 783us) server_negotiation.cc:299] Negotiation successful
0807 07:01:53.359320 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":248,"threads_started":1}
I20260807 07:01:53.360731 27895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:53.341178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:32885 (local address 127.21.83.2:38955)
0807 07:01:53.341941 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:53.341986 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:53.344420 (+ 2434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:53.345183 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:53.345191 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:53.345802 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:01:53.346386 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:53.346403 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:53.347776 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:53.347781 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:53.349108 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:53.349122 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:53.350045 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:53.350051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:53.350263 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:53.350905 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:01:53.350929 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:01:53.353130 (+ 2201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:01:53.356541 (+ 3411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:53.356550 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:53.356565 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:53.357096 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:53.357496 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:53.357500 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:53.357502 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:53.357722 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:53.358277 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:01:53.358285 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:01:53.358660 (+ 375us) client_negotiation.cc:769] Sending connection context
0807 07:01:53.358917 (+ 257us) client_negotiation.cc:240] Negotiation successful
0807 07:01:53.359506 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":374,"threads_started":1}
I20260807 07:01:53.362262 27890 heartbeater.cc:347] Connected to a master server at 127.21.83.62:32885
I20260807 07:01:53.362576 27890 heartbeater.cc:464] Registering TS with master...
I20260807 07:01:53.363325 27890 heartbeater.cc:511] Master 127.21.83.62:32885 requested a full tablet report, sending...
I20260807 07:01:53.364954 27393 ts_manager.cc:194] Registered new tserver with Master: 02bf35445aad4acaa61f6fe4dc300dc7 (127.21.83.2:43903)
I20260807 07:01:53.365832 27393 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.2@KRBTEST.COM'} at 127.21.83.2:38955
WARNING: no policy specified for kudu/127.21.83.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab.
Entry for principal kudu/127.21.83.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab.
I20260807 07:01:53.396080 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.3:0
--local_ip_for_outbound_sockets=127.21.83.3
--webserver_interface=127.21.83.3
--webserver_port=0
--tserver_master_addrs=127.21.83.62:32885
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab
--principal=kudu/127.21.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:34019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:01:53.731014 27899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:01:53.731314 27899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:01:53.731410 27899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:01:53.738888 27899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:01:53.738982 27899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:01:53.739102 27899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.3
I20260807 07:01:53.748551 27899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:34019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab
--principal=kudu/127.21.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.83.3
--webserver_port=0
--tserver_master_addrs=127.21.83.62:32885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.3
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:01:53.749799 27899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:01:53.751094 27899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:01:53.763801 27908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:53.763731 27915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:53.763756 27910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:01:53.764832 27899 server_base.cc:1034] running on GCE node
I20260807 07:01:53.765514 27899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:01:53.766534 27899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:01:53.767762 27899 hybrid_clock.cc:648] HybridClock initialized: now 1786086113767710 us; error 42 us; skew 500 ppm
I20260807 07:01:53.768121 27899 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:01:53 dist-test-slave-ncp9 krb5kdc[27322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086113, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:01:53.771418 27899 init.cc:377] Logged in from keytab as kudu/127.21.83.3@KRBTEST.COM (short username kudu)
I20260807 07:01:53.773032 27899 webserver.cc:460] Webserver started at http://127.21.83.3:37975/ using document root <none> and password file <none>
I20260807 07:01:53.773746 27899 fs_manager.cc:359] Metadata directory not provided
I20260807 07:01:53.773878 27899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:01:53.774204 27899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:01:53.777478 27899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/ts-2/data/instance:
uuid: "6bcf2576f4cf45dba488837d7e656fd9"
format_stamp: "Formatted at 2026-08-07 07:01:53 on dist-test-slave-ncp9"
server_key: "063ffed2466ea0c8bc57c1b5d58caac6"
server_key_iv: "ab08dc45f92dd711c76cce013f7aeebb"
server_key_version: "encryptionkey@0"
I20260807 07:01:53.778313 27899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/ts-2/wal/instance:
uuid: "6bcf2576f4cf45dba488837d7e656fd9"
format_stamp: "Formatted at 2026-08-07 07:01:53 on dist-test-slave-ncp9"
server_key: "063ffed2466ea0c8bc57c1b5d58caac6"
server_key_iv: "ab08dc45f92dd711c76cce013f7aeebb"
server_key_version: "encryptionkey@0"
I20260807 07:01:53.783820 27899 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260807 07:01:53.788507 27928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:53.790323 27899 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260807 07:01:53.790524 27899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/ts-2/wal
uuid: "6bcf2576f4cf45dba488837d7e656fd9"
format_stamp: "Formatted at 2026-08-07 07:01:53 on dist-test-slave-ncp9"
server_key: "063ffed2466ea0c8bc57c1b5d58caac6"
server_key_iv: "ab08dc45f92dd711c76cce013f7aeebb"
server_key_version: "encryptionkey@0"
I20260807 07:01:53.790824 27899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:01:53.814236 27899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:01:53.818677 27899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:01:53.818940 27899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:01:53.819928 27899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:01:53.821455 27899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:01:53.821530 27899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:53.821626 27899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:01:53.821671 27899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:53.825940 27671 heartbeater.cc:503] Master 127.21.83.62:32885 was elected leader, sending a full tablet report...
I20260807 07:01:53.854271 27899 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.3:35061
I20260807 07:01:53.854307 28092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.3:35061 every 8 connection(s)
I20260807 07:01:53.855597 27899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
I20260807 07:01:53.861117 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 27899
I20260807 07:01:53.861310 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786086078080656-21836-0/minicluster-data/ts-2/wal/instance
I20260807 07:01:53.861881 21836 external_mini_cluster.cc:1428] Setting key 2c15d4f86c448ae2967deb9fffa680ec
Aug 07 07:01:53 dist-test-slave-ncp9 krb5kdc[27322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086113, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.3@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:01:53.877197 27894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:53.858126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:49741 (local address 127.21.83.62:32885)
0807 07:01:53.858298 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:53.858303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:53.859449 (+ 1146us) server_negotiation.cc:407] Connection header received
0807 07:01:53.861774 (+ 2325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:53.861782 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:53.861900 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:53.861978 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:53.864631 (+ 2653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:53.865612 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:53.867196 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:53.867614 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:53.867720 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:53.871397 (+ 3677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:53.871402 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:53.871406 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:53.871442 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:53.874050 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:53.874775 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:53.874779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:53.874782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:53.874921 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:53.875344 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:53.875348 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:53.875350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:53.875666 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:53.875814 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:53.876611 (+ 797us) server_negotiation.cc:299] Negotiation successful
0807 07:01:53.876854 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 07:01:53.878298 28095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:53.858493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:32885 (local address 127.21.83.3:49741)
0807 07:01:53.859266 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:53.859308 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:53.861371 (+ 2063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:53.862886 (+ 1515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:53.862899 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:53.863680 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:01:53.864407 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:53.864436 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:53.865756 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:53.865760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:53.867037 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:53.867051 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:53.867869 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:53.867875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:53.868103 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:53.868887 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:01:53.868959 (+ 72us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:01:53.871167 (+ 2208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:01:53.874192 (+ 3025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:53.874199 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:53.874212 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:53.874651 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:53.875007 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:53.875010 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:53.875013 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:53.875243 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:53.875774 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:01:53.875779 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:01:53.876177 (+ 398us) client_negotiation.cc:769] Sending connection context
0807 07:01:53.876498 (+ 321us) client_negotiation.cc:240] Negotiation successful
0807 07:01:53.877042 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":302,"threads_started":1}
I20260807 07:01:53.879557 28093 heartbeater.cc:347] Connected to a master server at 127.21.83.62:32885
I20260807 07:01:53.879840 28093 heartbeater.cc:464] Registering TS with master...
I20260807 07:01:53.880549 28093 heartbeater.cc:511] Master 127.21.83.62:32885 requested a full tablet report, sending...
I20260807 07:01:53.881956 27393 ts_manager.cc:194] Registered new tserver with Master: 6bcf2576f4cf45dba488837d7e656fd9 (127.21.83.3:35061)
I20260807 07:01:53.882730 27393 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.3@KRBTEST.COM'} at 127.21.83.3:49741
I20260807 07:01:53.890141 21836 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 07:01:53 dist-test-slave-ncp9 krb5kdc[27322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 07:01:53 dist-test-slave-ncp9 krb5kdc[27322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.1@KRBTEST.COM
I20260807 07:01:53.947057 28102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:53.932733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60024 (local address 127.21.83.1:46699)
0807 07:01:53.933398 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:53.933406 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:53.933433 (+ 27us) server_negotiation.cc:407] Connection header received
0807 07:01:53.933496 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:53.933503 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:53.933713 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:53.933840 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:53.935168 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:53.936222 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:53.937489 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:53.938036 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:53.938133 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:53.941665 (+ 3532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:53.941677 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:53.941692 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:53.941725 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:53.944427 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:53.945303 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:53.945310 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:53.945312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:53.945442 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:53.945757 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:53.945760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:53.945762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:53.946084 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:53.946325 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:53.946503 (+ 178us) server_negotiation.cc:299] Negotiation successful
0807 07:01:53.946697 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":318,"threads_started":1}
Aug 07 07:01:53 dist-test-slave-ncp9 krb5kdc[27322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:01:53.964376 27894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:53.951459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54598 (local address 127.21.83.62:32885)
0807 07:01:53.951696 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:53.951701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:53.951817 (+ 116us) server_negotiation.cc:407] Connection header received
0807 07:01:53.951962 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:53.951966 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:53.952041 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:53.952117 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:53.953298 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:53.954009 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:53.955226 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:53.955616 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:53.955704 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:53.959472 (+ 3768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:53.959477 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:53.959481 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:53.959517 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:53.962009 (+ 2492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:53.962666 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:53.962669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:53.962671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:53.962798 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:53.963163 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:53.963166 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:53.963167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:53.963482 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:53.963679 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:53.963872 (+ 193us) server_negotiation.cc:299] Negotiation successful
0807 07:01:53.964046 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
W20260807 07:01:53.965082 27393 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54598
I20260807 07:01:53.966030 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 27482
I20260807 07:01:53.983573 28102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:53.967023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60028 (local address 127.21.83.1:46699)
0807 07:01:53.967258 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:53.967265 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:53.967370 (+ 105us) server_negotiation.cc:407] Connection header received
0807 07:01:53.967497 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:53.967502 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:53.967599 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:53.967679 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:53.968773 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:53.969488 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:53.970687 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:53.971127 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:53.971227 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:53.973654 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:53.973659 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:53.973663 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:53.973703 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:53.980574 (+ 6871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:53.981460 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:53.981464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:53.981467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:53.981634 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:53.982065 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:53.982069 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:53.982071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:53.982371 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:53.982570 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:53.983033 (+ 463us) server_negotiation.cc:299] Negotiation successful
0807 07:01:53.983247 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260807 07:01:53.984269 27663 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:01:54.069397 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 27482
I20260807 07:01:54.090760 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 27680
Aug 07 07:01:54 dist-test-slave-ncp9 krb5kdc[27322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.2@KRBTEST.COM
I20260807 07:01:54.106316 28108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:54.091593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38338 (local address 127.21.83.2:43903)
0807 07:01:54.092175 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:54.092184 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:54.092211 (+ 27us) server_negotiation.cc:407] Connection header received
0807 07:01:54.092274 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:54.092280 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:54.092485 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:54.092614 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:54.093933 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:54.094981 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:54.096233 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:54.096847 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:54.097102 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:54.100839 (+ 3737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:54.100850 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:54.100863 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:54.100937 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:54.103754 (+ 2817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:54.104357 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:54.104364 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:54.104366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:54.104496 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:54.104806 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:54.104809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:54.104811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:54.105280 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:54.105512 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:54.105738 (+ 226us) server_negotiation.cc:299] Negotiation successful
0807 07:01:54.105958 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":297,"threads_started":1}
I20260807 07:01:54.107589 27881 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:01:54.205603 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 27680
I20260807 07:01:54.228801 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 27899
Aug 07 07:01:54 dist-test-slave-ncp9 krb5kdc[27322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.3@KRBTEST.COM
I20260807 07:01:54.246402 28111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:54.229806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35258 (local address 127.21.83.3:35061)
0807 07:01:54.230446 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:54.230453 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:54.230481 (+ 28us) server_negotiation.cc:407] Connection header received
0807 07:01:54.230544 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:54.230551 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:54.230764 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:54.230914 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:54.232257 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:54.233568 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:54.234893 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:54.235472 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:54.235596 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:54.240119 (+ 4523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:54.240136 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:54.240153 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:54.240203 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:54.243200 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:54.244013 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:54.244021 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:54.244024 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:54.244183 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:54.244574 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:54.244578 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:54.244581 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:54.245049 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:54.245403 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:54.245748 (+ 345us) server_negotiation.cc:299] Negotiation successful
0807 07:01:54.246049 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":334,"threads_started":1}
I20260807 07:01:54.247808 28087 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:01:54.330750 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 27899
I20260807 07:01:54.353242 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 27341
I20260807 07:01:54.354431 27438 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:01:54.461901 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 27341
2026-08-07T07:01:54Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4156 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 07:01:54 dist-test-slave-ncp9 krb5kdc[28121](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 07:01:54 dist-test-slave-ncp9 krb5kdc[28121](info): set up 2 sockets
Aug 07 07:01:54 dist-test-slave-ncp9 krb5kdc[28121](info): commencing operation
krb5kdc: starting...
W20260807 07:01:55.671389 21836 mini_kdc.cc:121] Time spent starting KDC: real 1.162s user 0.001s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Aug 07 07:01:55 dist-test-slave-ncp9 krb5kdc[28121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T07:01:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:01:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
I20260807 07:01:55.908933 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:38267
--webserver_interface=127.21.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:45095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:38267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:01:56.240417 28135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:01:56.240738 28135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:01:56.240861 28135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:01:56.248263 28135 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:01:56.248373 28135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:01:56.248430 28135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:01:56.248513 28135 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:01:56.248582 28135 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:01:56.259017 28135 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:45095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:38267
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:38267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:01:56.260439 28135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:01:56.261893 28135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:01:56.273846 28142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:56.273738 28146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:56.273833 28144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:01:56.274917 28135 server_base.cc:1034] running on GCE node
I20260807 07:01:56.275712 28135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:01:56.277192 28135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:01:56.278399 28135 hybrid_clock.cc:648] HybridClock initialized: now 1786086116278355 us; error 32 us; skew 500 ppm
I20260807 07:01:56.278784 28135 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:01:56 dist-test-slave-ncp9 krb5kdc[28121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086116, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:01:56.282064 28135 init.cc:377] Logged in from keytab as kudu/127.21.83.62@KRBTEST.COM (short username kudu)
I20260807 07:01:56.283245 28135 fs_manager.cc:359] Metadata directory not provided
I20260807 07:01:56.283340 28135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:01:56.283638 28135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:01:56.286713 28135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "b06392ed002d4c82adb03efbb0cbb62e"
format_stamp: "Formatted at 2026-08-07 07:01:56 on dist-test-slave-ncp9"
server_key: "0c9479fe2696bfa04c91d1bfa3e1a1d3"
server_key_iv: "ca8b420ea26ed7cc4b8532d6d055a205"
server_key_version: "encryptionkey@0"
I20260807 07:01:56.287482 28135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "b06392ed002d4c82adb03efbb0cbb62e"
format_stamp: "Formatted at 2026-08-07 07:01:56 on dist-test-slave-ncp9"
server_key: "0c9479fe2696bfa04c91d1bfa3e1a1d3"
server_key_iv: "ca8b420ea26ed7cc4b8532d6d055a205"
server_key_version: "encryptionkey@0"
I20260807 07:01:56.292583 28135 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260807 07:01:56.295897 28155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:56.297185 28135 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 07:01:56.297353 28135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "b06392ed002d4c82adb03efbb0cbb62e"
format_stamp: "Formatted at 2026-08-07 07:01:56 on dist-test-slave-ncp9"
server_key: "0c9479fe2696bfa04c91d1bfa3e1a1d3"
server_key_iv: "ca8b420ea26ed7cc4b8532d6d055a205"
server_key_version: "encryptionkey@0"
I20260807 07:01:56.297545 28135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:01:56.312991 28135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:01:56.317225 28135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:01:56.317489 28135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:01:56.334553 28135 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.62:38267
I20260807 07:01:56.334573 28227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.62:38267 every 8 connection(s)
I20260807 07:01:56.335469 28135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:01:56.339129 28228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:01:56.341823 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 28135
I20260807 07:01:56.342036 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:01:56.342682 21836 external_mini_cluster.cc:1428] Setting key 26be53d40cbc958a66bbfb9589cb8bf9
I20260807 07:01:56.352295 28228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b06392ed002d4c82adb03efbb0cbb62e: Bootstrap starting.
I20260807 07:01:56.356215 28228 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b06392ed002d4c82adb03efbb0cbb62e: Neither blocks nor log segments found. Creating new log.
I20260807 07:01:56.357342 28228 log.cc:824] T 00000000000000000000000000000000 P b06392ed002d4c82adb03efbb0cbb62e: Log is configured to *not* fsync() on all Append() calls
Aug 07 07:01:56 dist-test-slave-ncp9 krb5kdc[28121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:01:56.360633 28228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b06392ed002d4c82adb03efbb0cbb62e: No bootstrap required, opened a new log
I20260807 07:01:56.367925 28232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:56.344683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55792 (local address 127.21.83.62:38267)
0807 07:01:56.345559 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:56.345575 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:56.345626 (+ 51us) server_negotiation.cc:407] Connection header received
0807 07:01:56.347861 (+ 2235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:56.347890 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:56.348410 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:56.348862 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:56.350285 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:56.352242 (+ 1957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:56.353736 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:56.354863 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:56.355063 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:56.359285 (+ 4222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:56.359296 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:56.359316 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:56.359369 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:56.363295 (+ 3926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:56.364041 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:56.364050 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:56.364059 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:56.364269 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:56.364624 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:56.364628 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:56.364632 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:56.365437 (+ 805us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:56.365745 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:56.366439 (+ 694us) server_negotiation.cc:299] Negotiation successful
0807 07:01:56.366849 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":386,"threads_started":1}
I20260807 07:01:56.370455 28228 raft_consensus.cc:348] T 00000000000000000000000000000000 P b06392ed002d4c82adb03efbb0cbb62e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b06392ed002d4c82adb03efbb0cbb62e" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 38267 } }
I20260807 07:01:56.370864 28228 raft_consensus.cc:374] T 00000000000000000000000000000000 P b06392ed002d4c82adb03efbb0cbb62e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:01:56.370983 28228 raft_consensus.cc:729] T 00000000000000000000000000000000 P b06392ed002d4c82adb03efbb0cbb62e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b06392ed002d4c82adb03efbb0cbb62e, State: Initialized, Role: FOLLOWER
I20260807 07:01:56.371498 28228 consensus_queue.cc:260] T 00000000000000000000000000000000 P b06392ed002d4c82adb03efbb0cbb62e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b06392ed002d4c82adb03efbb0cbb62e" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 38267 } }
I20260807 07:01:56.371712 28228 raft_consensus.cc:388] T 00000000000000000000000000000000 P b06392ed002d4c82adb03efbb0cbb62e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:01:56.371838 28228 raft_consensus.cc:482] T 00000000000000000000000000000000 P b06392ed002d4c82adb03efbb0cbb62e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:01:56.371989 28228 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b06392ed002d4c82adb03efbb0cbb62e [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:01:56.374222 28228 raft_consensus.cc:504] T 00000000000000000000000000000000 P b06392ed002d4c82adb03efbb0cbb62e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b06392ed002d4c82adb03efbb0cbb62e" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 38267 } }
I20260807 07:01:56.374679 28228 leader_election.cc:302] T 00000000000000000000000000000000 P b06392ed002d4c82adb03efbb0cbb62e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b06392ed002d4c82adb03efbb0cbb62e; no voters:
I20260807 07:01:56.375212 28228 leader_election.cc:288] T 00000000000000000000000000000000 P b06392ed002d4c82adb03efbb0cbb62e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 07:01:56.375370 28234 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b06392ed002d4c82adb03efbb0cbb62e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:01:56.377166 28234 raft_consensus.cc:686] T 00000000000000000000000000000000 P b06392ed002d4c82adb03efbb0cbb62e [term 1 LEADER]: Becoming Leader. State: Replica: b06392ed002d4c82adb03efbb0cbb62e, State: Running, Role: LEADER
I20260807 07:01:56.377311 28228 sys_catalog.cc:564] T 00000000000000000000000000000000 P b06392ed002d4c82adb03efbb0cbb62e [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:01:56.377645 28234 consensus_queue.cc:237] T 00000000000000000000000000000000 P b06392ed002d4c82adb03efbb0cbb62e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b06392ed002d4c82adb03efbb0cbb62e" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 38267 } }
I20260807 07:01:56.380396 28236 sys_catalog.cc:455] T 00000000000000000000000000000000 P b06392ed002d4c82adb03efbb0cbb62e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b06392ed002d4c82adb03efbb0cbb62e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b06392ed002d4c82adb03efbb0cbb62e" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 38267 } } }
I20260807 07:01:56.380808 28236 sys_catalog.cc:458] T 00000000000000000000000000000000 P b06392ed002d4c82adb03efbb0cbb62e [sys.catalog]: This master's current role is: LEADER
I20260807 07:01:56.380982 28238 sys_catalog.cc:455] T 00000000000000000000000000000000 P b06392ed002d4c82adb03efbb0cbb62e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b06392ed002d4c82adb03efbb0cbb62e. Latest consensus state: current_term: 1 leader_uuid: "b06392ed002d4c82adb03efbb0cbb62e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b06392ed002d4c82adb03efbb0cbb62e" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 38267 } } }
I20260807 07:01:56.381753 28238 sys_catalog.cc:458] T 00000000000000000000000000000000 P b06392ed002d4c82adb03efbb0cbb62e [sys.catalog]: This master's current role is: LEADER
I20260807 07:01:56.383666 28245 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:01:56.387591 28245 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:01:56.395057 28245 catalog_manager.cc:1347] Generated new cluster ID: dfdb4cd7b8da4ec998eda22be8962836
I20260807 07:01:56.395155 28245 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:01:56.418756 28245 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:01:56.420156 28245 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:01:56.429270 28245 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b06392ed002d4c82adb03efbb0cbb62e: Generated new TSK 0
I20260807 07:01:56.430042 28245 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.83.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab.
Entry for principal kudu/127.21.83.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab.
I20260807 07:01:56.486564 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.1:0
--local_ip_for_outbound_sockets=127.21.83.1
--webserver_interface=127.21.83.1
--webserver_port=0
--tserver_master_addrs=127.21.83.62:38267
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab
--principal=kudu/127.21.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:45095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:01:56.814185 28264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:01:56.814453 28264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:01:56.814601 28264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:01:56.821686 28264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:01:56.821792 28264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:01:56.821950 28264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.1
I20260807 07:01:56.831100 28264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:45095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab
--principal=kudu/127.21.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.83.1
--webserver_port=0
--tserver_master_addrs=127.21.83.62:38267
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.1
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:01:56.832274 28264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:01:56.833498 28264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:01:56.845841 28276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:56.845914 28273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:56.845914 28271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:01:56.847607 28264 server_base.cc:1034] running on GCE node
I20260807 07:01:56.848335 28264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:01:56.849424 28264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:01:56.850687 28264 hybrid_clock.cc:648] HybridClock initialized: now 1786086116850612 us; error 64 us; skew 500 ppm
I20260807 07:01:56.851066 28264 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:01:56 dist-test-slave-ncp9 krb5kdc[28121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086116, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:01:56.854550 28264 init.cc:377] Logged in from keytab as kudu/127.21.83.1@KRBTEST.COM (short username kudu)
I20260807 07:01:56.855772 28264 fs_manager.cc:359] Metadata directory not provided
I20260807 07:01:56.855876 28264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:01:56.856191 28264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:01:56.859439 28264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-0/data/instance:
uuid: "0ff693488f35473aab7e986e1c724020"
format_stamp: "Formatted at 2026-08-07 07:01:56 on dist-test-slave-ncp9"
server_key: "82668b8c212295686063213442da08ed"
server_key_iv: "2d1285ae63819b34f085d796434c8024"
server_key_version: "encryptionkey@0"
I20260807 07:01:56.860359 28264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-0/wal/instance:
uuid: "0ff693488f35473aab7e986e1c724020"
format_stamp: "Formatted at 2026-08-07 07:01:56 on dist-test-slave-ncp9"
server_key: "82668b8c212295686063213442da08ed"
server_key_iv: "2d1285ae63819b34f085d796434c8024"
server_key_version: "encryptionkey@0"
I20260807 07:01:56.866098 28264 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260807 07:01:56.869953 28283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:56.871371 28264 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260807 07:01:56.871541 28264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-0/wal
uuid: "0ff693488f35473aab7e986e1c724020"
format_stamp: "Formatted at 2026-08-07 07:01:56 on dist-test-slave-ncp9"
server_key: "82668b8c212295686063213442da08ed"
server_key_iv: "2d1285ae63819b34f085d796434c8024"
server_key_version: "encryptionkey@0"
I20260807 07:01:56.871750 28264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:01:56.896796 28264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:01:56.901051 28264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:01:56.901292 28264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:01:56.902247 28264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:01:56.903568 28264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:01:56.903659 28264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:56.903738 28264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:01:56.903791 28264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:56.934528 28264 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.1:41029
I20260807 07:01:56.934587 28444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.1:41029 every 8 connection(s)
I20260807 07:01:56.935497 28264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
I20260807 07:01:56.941592 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 28264
I20260807 07:01:56.941828 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-0/wal/instance
I20260807 07:01:56.942481 21836 external_mini_cluster.cc:1428] Setting key a84ca1a60b08bf424a490b1e68f022c7
Aug 07 07:01:56 dist-test-slave-ncp9 krb5kdc[28121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086116, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.1@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:01:56.956959 28447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:56.938373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:60647 (local address 127.21.83.62:38267)
0807 07:01:56.938867 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:56.938872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:56.939844 (+ 972us) server_negotiation.cc:407] Connection header received
0807 07:01:56.941824 (+ 1980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:56.941830 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:56.941928 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:56.942042 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:56.943884 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:56.944748 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:56.946600 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:56.947030 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:56.947167 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:56.951088 (+ 3921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:56.951094 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:56.951098 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:56.951144 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:56.953653 (+ 2509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:56.954491 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:56.954495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:56.954497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:56.954630 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:56.955076 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:56.955080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:56.955083 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:56.955378 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:56.955520 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:56.956296 (+ 776us) server_negotiation.cc:299] Negotiation successful
0807 07:01:56.956551 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":243,"threads_started":1}
I20260807 07:01:56.958022 28449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:56.938933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38267 (local address 127.21.83.1:60647)
0807 07:01:56.939654 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:56.939697 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:56.941511 (+ 1814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:56.942346 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:56.942355 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:56.942949 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:01:56.943706 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:56.943728 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:56.944953 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:56.944958 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:56.946412 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:56.946429 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:56.947292 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:56.947300 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:56.947532 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:56.948264 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:01:56.948288 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:01:56.950823 (+ 2535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:01:56.953837 (+ 3014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:56.953846 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:56.953862 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:56.954344 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:56.954742 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:56.954748 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:56.954751 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:56.954955 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:56.955512 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:01:56.955521 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:01:56.955902 (+ 381us) client_negotiation.cc:769] Sending connection context
0807 07:01:56.956149 (+ 247us) client_negotiation.cc:240] Negotiation successful
0807 07:01:56.956760 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":294,"threads_started":1}
I20260807 07:01:56.959765 28445 heartbeater.cc:347] Connected to a master server at 127.21.83.62:38267
I20260807 07:01:56.960070 28445 heartbeater.cc:464] Registering TS with master...
I20260807 07:01:56.960754 28445 heartbeater.cc:511] Master 127.21.83.62:38267 requested a full tablet report, sending...
I20260807 07:01:56.963316 28177 ts_manager.cc:194] Registered new tserver with Master: 0ff693488f35473aab7e986e1c724020 (127.21.83.1:41029)
I20260807 07:01:56.965525 28177 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.1@KRBTEST.COM'} at 127.21.83.1:60647
WARNING: no policy specified for kudu/127.21.83.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab.
Entry for principal kudu/127.21.83.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab.
I20260807 07:01:56.997640 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.2:0
--local_ip_for_outbound_sockets=127.21.83.2
--webserver_interface=127.21.83.2
--webserver_port=0
--tserver_master_addrs=127.21.83.62:38267
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab
--principal=kudu/127.21.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:45095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:01:57.361317 28454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:01:57.361979 28454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:01:57.362320 28454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:01:57.375974 28454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:01:57.376240 28454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:01:57.376581 28454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.2
I20260807 07:01:57.396221 28454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:45095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab
--principal=kudu/127.21.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.83.2
--webserver_port=0
--tserver_master_addrs=127.21.83.62:38267
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.2
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:01:57.398934 28454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:01:57.401690 28454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:01:57.425952 28461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:57.426021 28468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:57.426913 28460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:01:57.428391 28454 server_base.cc:1034] running on GCE node
I20260807 07:01:57.429554 28454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:01:57.431183 28454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:01:57.432545 28454 hybrid_clock.cc:648] HybridClock initialized: now 1786086117432469 us; error 68 us; skew 500 ppm
I20260807 07:01:57.433096 28454 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:01:57 dist-test-slave-ncp9 krb5kdc[28121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086117, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:01:57.445178 28454 init.cc:377] Logged in from keytab as kudu/127.21.83.2@KRBTEST.COM (short username kudu)
I20260807 07:01:57.448225 28454 fs_manager.cc:359] Metadata directory not provided
I20260807 07:01:57.448673 28454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:01:57.449402 28454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:01:57.457208 28454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-1/data/instance:
uuid: "8e76949275b94888adef92c64b34d096"
format_stamp: "Formatted at 2026-08-07 07:01:57 on dist-test-slave-ncp9"
server_key: "df9750d21310b5094837edd57d2bc650"
server_key_iv: "1e6558c3af0c16898b23074db1ef278d"
server_key_version: "encryptionkey@0"
I20260807 07:01:57.458878 28454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-1/wal/instance:
uuid: "8e76949275b94888adef92c64b34d096"
format_stamp: "Formatted at 2026-08-07 07:01:57 on dist-test-slave-ncp9"
server_key: "df9750d21310b5094837edd57d2bc650"
server_key_iv: "1e6558c3af0c16898b23074db1ef278d"
server_key_version: "encryptionkey@0"
I20260807 07:01:57.469264 28454 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.011s sys 0.001s
I20260807 07:01:57.475656 28477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:57.478273 28454 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260807 07:01:57.478590 28454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-1/wal
uuid: "8e76949275b94888adef92c64b34d096"
format_stamp: "Formatted at 2026-08-07 07:01:57 on dist-test-slave-ncp9"
server_key: "df9750d21310b5094837edd57d2bc650"
server_key_iv: "1e6558c3af0c16898b23074db1ef278d"
server_key_version: "encryptionkey@0"
I20260807 07:01:57.478971 28454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:01:57.509407 28454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:01:57.516016 28454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:01:57.516426 28454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:01:57.518054 28454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:01:57.520354 28454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:01:57.520511 28454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:57.520699 28454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:01:57.520812 28454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:57.573756 28454 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.2:38737
I20260807 07:01:57.573865 28651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.2:38737 every 8 connection(s)
I20260807 07:01:57.575368 28454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
I20260807 07:01:57.585417 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 28454
I20260807 07:01:57.585835 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-1/wal/instance
I20260807 07:01:57.586853 21836 external_mini_cluster.cc:1428] Setting key f5bd7af8393a9f23621dc7ff5701ec7a
Aug 07 07:01:57 dist-test-slave-ncp9 krb5kdc[28121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086117, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.2@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:01:57.619102 28655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:57.579927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:34029 (local address 127.21.83.62:38267)
0807 07:01:57.580756 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:57.580769 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:57.582551 (+ 1782us) server_negotiation.cc:407] Connection header received
0807 07:01:57.586576 (+ 4025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:57.586583 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:57.586751 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:57.586984 (+ 233us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:57.592966 (+ 5982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:57.594962 (+ 1996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:57.598211 (+ 3249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:57.599401 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:57.599671 (+ 270us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:57.607284 (+ 7613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:57.607299 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:57.607304 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:57.607376 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:57.613119 (+ 5743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:57.614451 (+ 1332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:57.614458 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:57.614473 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:57.614758 (+ 285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:57.615659 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:57.615665 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:57.615669 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:57.616479 (+ 810us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:57.616766 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:57.617985 (+ 1219us) server_negotiation.cc:299] Negotiation successful
0807 07:01:57.618419 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":459,"threads_started":1}
I20260807 07:01:57.620258 28656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:57.580981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38267 (local address 127.21.83.2:34029)
0807 07:01:57.582273 (+ 1292us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:57.582342 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:57.585896 (+ 3554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:57.587570 (+ 1674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:57.587587 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:57.590607 (+ 3020us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:01:57.592505 (+ 1898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:57.592579 (+ 74us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:57.595295 (+ 2716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:57.595301 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:57.597875 (+ 2574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:57.597903 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:57.599816 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:57.599831 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:57.600188 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:57.601604 (+ 1416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:01:57.601677 (+ 73us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:01:57.606842 (+ 5165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:01:57.613343 (+ 6501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:57.613353 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:57.613368 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:57.614209 (+ 841us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:57.614959 (+ 750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:57.614965 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:57.614968 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:57.615434 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:57.616720 (+ 1286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:01:57.616736 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:01:57.617358 (+ 622us) client_negotiation.cc:769] Sending connection context
0807 07:01:57.617761 (+ 403us) client_negotiation.cc:240] Negotiation successful
0807 07:01:57.618445 (+ 684us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":857,"thread_start_us":485,"threads_started":1}
I20260807 07:01:57.622668 28653 heartbeater.cc:347] Connected to a master server at 127.21.83.62:38267
I20260807 07:01:57.623343 28653 heartbeater.cc:464] Registering TS with master...
I20260807 07:01:57.624866 28653 heartbeater.cc:511] Master 127.21.83.62:38267 requested a full tablet report, sending...
I20260807 07:01:57.627635 28177 ts_manager.cc:194] Registered new tserver with Master: 8e76949275b94888adef92c64b34d096 (127.21.83.2:38737)
I20260807 07:01:57.629025 28177 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.2@KRBTEST.COM'} at 127.21.83.2:34029
WARNING: no policy specified for kudu/127.21.83.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab.
Entry for principal kudu/127.21.83.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab.
I20260807 07:01:57.677268 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.3:0
--local_ip_for_outbound_sockets=127.21.83.3
--webserver_interface=127.21.83.3
--webserver_port=0
--tserver_master_addrs=127.21.83.62:38267
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab
--principal=kudu/127.21.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:45095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 07:01:57.969061 28445 heartbeater.cc:503] Master 127.21.83.62:38267 was elected leader, sending a full tablet report...
W20260807 07:01:58.039747 28660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:01:58.039994 28660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:01:58.040128 28660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:01:58.047067 28660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:01:58.047159 28660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:01:58.047322 28660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.3
I20260807 07:01:58.056918 28660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:45095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab
--principal=kudu/127.21.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.83.3
--webserver_port=0
--tserver_master_addrs=127.21.83.62:38267
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.3
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:01:58.058264 28660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:01:58.059572 28660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:01:58.071537 28666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:58.071570 28673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:58.071606 28670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:01:58.072629 28660 server_base.cc:1034] running on GCE node
I20260807 07:01:58.073189 28660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:01:58.073982 28660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:01:58.075203 28660 hybrid_clock.cc:648] HybridClock initialized: now 1786086118075148 us; error 43 us; skew 500 ppm
I20260807 07:01:58.075510 28660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:01:58 dist-test-slave-ncp9 krb5kdc[28121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086118, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:01:58.078629 28660 init.cc:377] Logged in from keytab as kudu/127.21.83.3@KRBTEST.COM (short username kudu)
I20260807 07:01:58.079671 28660 fs_manager.cc:359] Metadata directory not provided
I20260807 07:01:58.079772 28660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:01:58.080085 28660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:01:58.082927 28660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-2/data/instance:
uuid: "98275b615c5949849d981d1518ee01a1"
format_stamp: "Formatted at 2026-08-07 07:01:58 on dist-test-slave-ncp9"
server_key: "863975224fbcfd78c738170a030ad51d"
server_key_iv: "b301b4082a949819cd276d1eab0ae322"
server_key_version: "encryptionkey@0"
I20260807 07:01:58.083693 28660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-2/wal/instance:
uuid: "98275b615c5949849d981d1518ee01a1"
format_stamp: "Formatted at 2026-08-07 07:01:58 on dist-test-slave-ncp9"
server_key: "863975224fbcfd78c738170a030ad51d"
server_key_iv: "b301b4082a949819cd276d1eab0ae322"
server_key_version: "encryptionkey@0"
I20260807 07:01:58.088420 28660 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260807 07:01:58.091629 28680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:58.092767 28660 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 07:01:58.092978 28660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-2/wal
uuid: "98275b615c5949849d981d1518ee01a1"
format_stamp: "Formatted at 2026-08-07 07:01:58 on dist-test-slave-ncp9"
server_key: "863975224fbcfd78c738170a030ad51d"
server_key_iv: "b301b4082a949819cd276d1eab0ae322"
server_key_version: "encryptionkey@0"
I20260807 07:01:58.093192 28660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:01:58.106843 28660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:01:58.110973 28660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:01:58.111245 28660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:01:58.112223 28660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:01:58.113538 28660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:01:58.113642 28660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:58.113750 28660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:01:58.113821 28660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:58.143927 28660 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.3:46765
I20260807 07:01:58.143976 28834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.3:46765 every 8 connection(s)
I20260807 07:01:58.144717 28660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
I20260807 07:01:58.149260 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 28660
I20260807 07:01:58.149431 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-2/wal/instance
I20260807 07:01:58.149837 21836 external_mini_cluster.cc:1428] Setting key ac135f086596d752ed123d202920ff37
Aug 07 07:01:58 dist-test-slave-ncp9 krb5kdc[28121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086118, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.3@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:01:58.164048 28838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:58.147125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:49287 (local address 127.21.83.62:38267)
0807 07:01:58.147600 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:58.147606 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:58.148529 (+ 923us) server_negotiation.cc:407] Connection header received
0807 07:01:58.150462 (+ 1933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:58.150468 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:58.150552 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:58.150647 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:58.152165 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:58.153007 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:58.154556 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:58.155020 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:58.155133 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:58.158561 (+ 3428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:58.158567 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:58.158570 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:58.158611 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:58.161006 (+ 2395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:58.161744 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:58.161749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:58.161754 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:58.161878 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:58.162310 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:58.162315 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:58.162318 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:58.162579 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:58.162730 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:58.163412 (+ 682us) server_negotiation.cc:299] Negotiation successful
0807 07:01:58.163649 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":207,"threads_started":1}
I20260807 07:01:58.165053 28839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:58.147634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38267 (local address 127.21.83.3:49287)
0807 07:01:58.148364 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:58.148409 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:58.150172 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:58.150895 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:58.150904 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:58.151419 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:01:58.151970 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:58.151989 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:58.153162 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:58.153166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:58.154405 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:58.154419 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:58.155263 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:58.155271 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:58.155484 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:58.156124 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:01:58.156152 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:01:58.158343 (+ 2191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:01:58.161142 (+ 2799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:58.161151 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:58.161164 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:58.161622 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:58.161985 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:01:58.161989 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:01:58.161993 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:01:58.162171 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:01:58.162717 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:01:58.162724 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:01:58.163040 (+ 316us) client_negotiation.cc:769] Sending connection context
0807 07:01:58.163292 (+ 252us) client_negotiation.cc:240] Negotiation successful
0807 07:01:58.163792 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":298,"threads_started":1}
I20260807 07:01:58.166289 28836 heartbeater.cc:347] Connected to a master server at 127.21.83.62:38267
I20260807 07:01:58.166590 28836 heartbeater.cc:464] Registering TS with master...
I20260807 07:01:58.167223 28836 heartbeater.cc:511] Master 127.21.83.62:38267 requested a full tablet report, sending...
I20260807 07:01:58.168490 28177 ts_manager.cc:194] Registered new tserver with Master: 98275b615c5949849d981d1518ee01a1 (127.21.83.3:46765)
I20260807 07:01:58.169292 28177 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.3@KRBTEST.COM'} at 127.21.83.3:49287
I20260807 07:01:58.176533 21836 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 07:01:58.192188 28838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:58.180974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55806 (local address 127.21.83.62:38267)
0807 07:01:58.181148 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:58.181155 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:58.181606 (+ 451us) server_negotiation.cc:407] Connection header received
0807 07:01:58.181763 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:58.181767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:58.181825 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:58.181900 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:58.183263 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:58.183955 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:58.185139 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:58.185528 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:58.185613 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:58.187697 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:58.187702 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:58.187705 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:58.187733 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:58.190118 (+ 2385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:58.190660 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:58.190663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:58.190665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:58.190771 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:58.191038 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:58.191041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:58.191042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:58.191314 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:58.191484 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:58.191726 (+ 242us) server_negotiation.cc:299] Negotiation successful
0807 07:01:58.191895 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260807 07:01:58.195533 28177 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55806:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 07:01:58.202340 28177 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 07:01:58.225670 28851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:58.213112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.3:46765 (local address 127.0.0.1:34312)
0807 07:01:58.214465 (+ 1353us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:58.214509 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:58.214663 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:58.218405 (+ 3742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:58.218412 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:58.218441 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:01:58.218882 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:58.218895 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:58.220976 (+ 2081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:58.220981 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:58.222833 (+ 1852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:58.222845 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:58.224668 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:58.224674 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:58.224856 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:58.224936 (+ 80us) client_negotiation.cc:769] Sending connection context
0807 07:01:58.225064 (+ 128us) client_negotiation.cc:240] Negotiation successful
0807 07:01:58.225245 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1130,"mutex_wait_us":30,"thread_start_us":984,"threads_started":1}
I20260807 07:01:58.226055 28857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:58.215312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56762 (local address 127.21.83.2:38737)
0807 07:01:58.217328 (+ 2016us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:58.217336 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:58.218013 (+ 677us) server_negotiation.cc:407] Connection header received
0807 07:01:58.218106 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:58.218112 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:58.218293 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:58.218453 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:01:58.219217 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:58.220496 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:58.223286 (+ 2790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:58.224671 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:58.224796 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:58.225024 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:58.225567 (+ 543us) server_negotiation.cc:299] Negotiation successful
0807 07:01:58.225739 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1862,"thread_start_us":1766,"threads_started":1}
I20260807 07:01:58.226263 28855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:58.213625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:41029 (local address 127.0.0.1:53632)
0807 07:01:58.215301 (+ 1676us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:58.215318 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:58.215429 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:58.219610 (+ 4181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:58.219614 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:58.219639 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:01:58.219976 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:58.219986 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:58.221832 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:58.221837 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:58.223714 (+ 1877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:58.223729 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:58.225443 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:58.225450 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:58.225619 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:58.225651 (+ 32us) client_negotiation.cc:769] Sending connection context
0807 07:01:58.225740 (+ 89us) client_negotiation.cc:240] Negotiation successful
0807 07:01:58.225858 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1565,"mutex_wait_us":1,"thread_start_us":3866,"threads_started":1}
I20260807 07:01:58.226739 28854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:58.214279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53632 (local address 127.21.83.1:41029)
0807 07:01:58.219094 (+ 4815us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:58.219102 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:58.219126 (+ 24us) server_negotiation.cc:407] Connection header received
0807 07:01:58.219220 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:58.219227 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:58.219417 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:58.219611 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:01:58.220140 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:58.221670 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:58.223894 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:58.225235 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:58.225350 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:58.225564 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:58.226026 (+ 462us) server_negotiation.cc:299] Negotiation successful
0807 07:01:58.226510 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4643,"thread_start_us":4507,"threads_started":1}
I20260807 07:01:58.226773 28853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:58.214820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.2:38737 (local address 127.0.0.1:56762)
0807 07:01:58.217743 (+ 2923us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:58.217759 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:58.217871 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:58.218534 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:58.218539 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:58.218574 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:01:58.219059 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:58.219071 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:58.220657 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:58.220662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:58.223118 (+ 2456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:58.223133 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:58.225163 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:58.225170 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:58.225338 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:58.225379 (+ 41us) client_negotiation.cc:769] Sending connection context
0807 07:01:58.226306 (+ 927us) client_negotiation.cc:240] Negotiation successful
0807 07:01:58.226431 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2785,"mutex_wait_us":71,"thread_start_us":680,"threads_started":1}
I20260807 07:01:58.226925 28852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:58.213362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34312 (local address 127.21.83.3:46765)
0807 07:01:58.217803 (+ 4441us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:58.217811 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:58.217842 (+ 31us) server_negotiation.cc:407] Connection header received
0807 07:01:58.217942 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:58.217950 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:58.218172 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:58.219119 (+ 947us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:01:58.219406 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:58.220802 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:58.222989 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:58.224463 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:58.226152 (+ 1689us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:58.226388 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:58.226457 (+ 69us) server_negotiation.cc:299] Negotiation successful
0807 07:01:58.226605 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4281,"thread_start_us":419,"threads_started":1}
I20260807 07:01:58.230093 28553 tablet_service.cc:1467] Processing CreateTablet for tablet deea7403c5044ee2955f22a567e78df8 (DEFAULT_TABLE table=test-table [id=e7115d46d18b4e54b22a714d710eed1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:01:58.230252 28744 tablet_service.cc:1467] Processing CreateTablet for tablet deea7403c5044ee2955f22a567e78df8 (DEFAULT_TABLE table=test-table [id=e7115d46d18b4e54b22a714d710eed1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:01:58.230245 28358 tablet_service.cc:1467] Processing CreateTablet for tablet deea7403c5044ee2955f22a567e78df8 (DEFAULT_TABLE table=test-table [id=e7115d46d18b4e54b22a714d710eed1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:01:58.231220 28553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet deea7403c5044ee2955f22a567e78df8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:01:58.231549 28744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet deea7403c5044ee2955f22a567e78df8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:01:58.231544 28358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet deea7403c5044ee2955f22a567e78df8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:01:58.241917 28859 tablet_bootstrap.cc:492] T deea7403c5044ee2955f22a567e78df8 P 8e76949275b94888adef92c64b34d096: Bootstrap starting.
I20260807 07:01:58.244813 28861 tablet_bootstrap.cc:492] T deea7403c5044ee2955f22a567e78df8 P 0ff693488f35473aab7e986e1c724020: Bootstrap starting.
I20260807 07:01:58.246179 28859 tablet_bootstrap.cc:654] T deea7403c5044ee2955f22a567e78df8 P 8e76949275b94888adef92c64b34d096: Neither blocks nor log segments found. Creating new log.
I20260807 07:01:58.247706 28859 log.cc:824] T deea7403c5044ee2955f22a567e78df8 P 8e76949275b94888adef92c64b34d096: Log is configured to *not* fsync() on all Append() calls
I20260807 07:01:58.247938 28861 tablet_bootstrap.cc:654] T deea7403c5044ee2955f22a567e78df8 P 0ff693488f35473aab7e986e1c724020: Neither blocks nor log segments found. Creating new log.
I20260807 07:01:58.249161 28861 log.cc:824] T deea7403c5044ee2955f22a567e78df8 P 0ff693488f35473aab7e986e1c724020: Log is configured to *not* fsync() on all Append() calls
I20260807 07:01:58.249779 28862 tablet_bootstrap.cc:492] T deea7403c5044ee2955f22a567e78df8 P 98275b615c5949849d981d1518ee01a1: Bootstrap starting.
I20260807 07:01:58.250946 28859 tablet_bootstrap.cc:492] T deea7403c5044ee2955f22a567e78df8 P 8e76949275b94888adef92c64b34d096: No bootstrap required, opened a new log
I20260807 07:01:58.251240 28859 ts_tablet_manager.cc:1397] T deea7403c5044ee2955f22a567e78df8 P 8e76949275b94888adef92c64b34d096: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.006s
I20260807 07:01:58.253226 28862 tablet_bootstrap.cc:654] T deea7403c5044ee2955f22a567e78df8 P 98275b615c5949849d981d1518ee01a1: Neither blocks nor log segments found. Creating new log.
I20260807 07:01:58.254347 28862 log.cc:824] T deea7403c5044ee2955f22a567e78df8 P 98275b615c5949849d981d1518ee01a1: Log is configured to *not* fsync() on all Append() calls
I20260807 07:01:58.255348 28861 tablet_bootstrap.cc:492] T deea7403c5044ee2955f22a567e78df8 P 0ff693488f35473aab7e986e1c724020: No bootstrap required, opened a new log
I20260807 07:01:58.255667 28861 ts_tablet_manager.cc:1397] T deea7403c5044ee2955f22a567e78df8 P 0ff693488f35473aab7e986e1c724020: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.010s
I20260807 07:01:58.257694 28862 tablet_bootstrap.cc:492] T deea7403c5044ee2955f22a567e78df8 P 98275b615c5949849d981d1518ee01a1: No bootstrap required, opened a new log
I20260807 07:01:58.257938 28862 ts_tablet_manager.cc:1397] T deea7403c5044ee2955f22a567e78df8 P 98275b615c5949849d981d1518ee01a1: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260807 07:01:58.260957 28859 raft_consensus.cc:348] T deea7403c5044ee2955f22a567e78df8 P 8e76949275b94888adef92c64b34d096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98275b615c5949849d981d1518ee01a1" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 46765 } } peers { permanent_uuid: "0ff693488f35473aab7e986e1c724020" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 41029 } } peers { permanent_uuid: "8e76949275b94888adef92c64b34d096" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 38737 } }
I20260807 07:01:58.261371 28859 raft_consensus.cc:374] T deea7403c5044ee2955f22a567e78df8 P 8e76949275b94888adef92c64b34d096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:01:58.261475 28859 raft_consensus.cc:729] T deea7403c5044ee2955f22a567e78df8 P 8e76949275b94888adef92c64b34d096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e76949275b94888adef92c64b34d096, State: Initialized, Role: FOLLOWER
I20260807 07:01:58.262081 28859 consensus_queue.cc:260] T deea7403c5044ee2955f22a567e78df8 P 8e76949275b94888adef92c64b34d096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98275b615c5949849d981d1518ee01a1" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 46765 } } peers { permanent_uuid: "0ff693488f35473aab7e986e1c724020" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 41029 } } peers { permanent_uuid: "8e76949275b94888adef92c64b34d096" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 38737 } }
I20260807 07:01:58.263300 28653 heartbeater.cc:503] Master 127.21.83.62:38267 was elected leader, sending a full tablet report...
I20260807 07:01:58.263903 28859 ts_tablet_manager.cc:1428] T deea7403c5044ee2955f22a567e78df8 P 8e76949275b94888adef92c64b34d096: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260807 07:01:58.262792 28861 raft_consensus.cc:348] T deea7403c5044ee2955f22a567e78df8 P 0ff693488f35473aab7e986e1c724020 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98275b615c5949849d981d1518ee01a1" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 46765 } } peers { permanent_uuid: "0ff693488f35473aab7e986e1c724020" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 41029 } } peers { permanent_uuid: "8e76949275b94888adef92c64b34d096" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 38737 } }
I20260807 07:01:58.264187 28861 raft_consensus.cc:374] T deea7403c5044ee2955f22a567e78df8 P 0ff693488f35473aab7e986e1c724020 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:01:58.264307 28861 raft_consensus.cc:729] T deea7403c5044ee2955f22a567e78df8 P 0ff693488f35473aab7e986e1c724020 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ff693488f35473aab7e986e1c724020, State: Initialized, Role: FOLLOWER
I20260807 07:01:58.265084 28861 consensus_queue.cc:260] T deea7403c5044ee2955f22a567e78df8 P 0ff693488f35473aab7e986e1c724020 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98275b615c5949849d981d1518ee01a1" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 46765 } } peers { permanent_uuid: "0ff693488f35473aab7e986e1c724020" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 41029 } } peers { permanent_uuid: "8e76949275b94888adef92c64b34d096" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 38737 } }
I20260807 07:01:58.268399 28861 ts_tablet_manager.cc:1428] T deea7403c5044ee2955f22a567e78df8 P 0ff693488f35473aab7e986e1c724020: Time spent starting tablet: real 0.013s user 0.007s sys 0.004s
I20260807 07:01:58.268225 28862 raft_consensus.cc:348] T deea7403c5044ee2955f22a567e78df8 P 98275b615c5949849d981d1518ee01a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98275b615c5949849d981d1518ee01a1" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 46765 } } peers { permanent_uuid: "0ff693488f35473aab7e986e1c724020" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 41029 } } peers { permanent_uuid: "8e76949275b94888adef92c64b34d096" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 38737 } }
I20260807 07:01:58.268594 28862 raft_consensus.cc:374] T deea7403c5044ee2955f22a567e78df8 P 98275b615c5949849d981d1518ee01a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:01:58.268682 28862 raft_consensus.cc:729] T deea7403c5044ee2955f22a567e78df8 P 98275b615c5949849d981d1518ee01a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98275b615c5949849d981d1518ee01a1, State: Initialized, Role: FOLLOWER
I20260807 07:01:58.269248 28862 consensus_queue.cc:260] T deea7403c5044ee2955f22a567e78df8 P 98275b615c5949849d981d1518ee01a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98275b615c5949849d981d1518ee01a1" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 46765 } } peers { permanent_uuid: "0ff693488f35473aab7e986e1c724020" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 41029 } } peers { permanent_uuid: "8e76949275b94888adef92c64b34d096" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 38737 } }
I20260807 07:01:58.270396 28836 heartbeater.cc:503] Master 127.21.83.62:38267 was elected leader, sending a full tablet report...
I20260807 07:01:58.270886 28862 ts_tablet_manager.cc:1428] T deea7403c5044ee2955f22a567e78df8 P 98275b615c5949849d981d1518ee01a1: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260807 07:01:58.306751 28869 raft_consensus.cc:482] T deea7403c5044ee2955f22a567e78df8 P 98275b615c5949849d981d1518ee01a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:01:58.307010 28869 raft_consensus.cc:504] T deea7403c5044ee2955f22a567e78df8 P 98275b615c5949849d981d1518ee01a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98275b615c5949849d981d1518ee01a1" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 46765 } } peers { permanent_uuid: "0ff693488f35473aab7e986e1c724020" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 41029 } } peers { permanent_uuid: "8e76949275b94888adef92c64b34d096" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 38737 } }
I20260807 07:01:58.308568 28869 leader_election.cc:288] T deea7403c5044ee2955f22a567e78df8 P 98275b615c5949849d981d1518ee01a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ff693488f35473aab7e986e1c724020 (127.21.83.1:41029), 8e76949275b94888adef92c64b34d096 (127.21.83.2:38737)
I20260807 07:01:58.314321 28839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:58.308930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.2:38737 (local address 127.21.83.3:57941)
0807 07:01:58.309239 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:58.309259 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:58.309387 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:58.309929 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:58.309934 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:58.309960 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:01:58.310126 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:58.310134 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:58.311087 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:58.311092 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:58.312422 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:58.312434 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:58.313783 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:58.313787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:58.313923 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:58.313948 (+ 25us) client_negotiation.cc:769] Sending connection context
0807 07:01:58.314022 (+ 74us) client_negotiation.cc:240] Negotiation successful
0807 07:01:58.314108 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":38}
I20260807 07:01:58.314430 28857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:58.309029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:57941 (local address 127.21.83.2:38737)
0807 07:01:58.309671 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:58.309675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:58.309692 (+ 17us) server_negotiation.cc:407] Connection header received
0807 07:01:58.309748 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:58.309751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:58.309807 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:58.309888 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:01:58.310262 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:58.310955 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:58.312576 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:58.313640 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:58.313769 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:58.313861 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:58.314062 (+ 201us) server_negotiation.cc:299] Negotiation successful
0807 07:01:58.314187 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526}
I20260807 07:01:58.315135 28854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:58.308977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:47471 (local address 127.21.83.1:41029)
0807 07:01:58.309649 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:58.309655 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:58.309759 (+ 104us) server_negotiation.cc:407] Connection header received
0807 07:01:58.309918 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:58.309922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:58.309979 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:58.310049 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:01:58.310583 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:58.311295 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:58.313181 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:58.314236 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:58.314363 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:58.314455 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:58.314798 (+ 343us) server_negotiation.cc:299] Negotiation successful
0807 07:01:58.314914 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533}
I20260807 07:01:58.315150 28870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:58.308949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:41029 (local address 127.21.83.3:47471)
0807 07:01:58.309663 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0807 07:01:58.309689 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 07:01:58.309804 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:01:58.310093 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:01:58.310098 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:01:58.310122 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:01:58.310426 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:58.310434 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:58.311439 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:58.311443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:58.313014 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:01:58.313029 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:01:58.314457 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:01:58.314463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:01:58.314629 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:58.314661 (+ 32us) client_negotiation.cc:769] Sending connection context
0807 07:01:58.314764 (+ 103us) client_negotiation.cc:240] Negotiation successful
0807 07:01:58.314880 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"mutex_wait_us":26,"thread_start_us":366,"threads_started":1}
I20260807 07:01:58.315333 28589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deea7403c5044ee2955f22a567e78df8" candidate_uuid: "98275b615c5949849d981d1518ee01a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e76949275b94888adef92c64b34d096" is_pre_election: true
I20260807 07:01:58.315845 28589 raft_consensus.cc:2454] T deea7403c5044ee2955f22a567e78df8 P 8e76949275b94888adef92c64b34d096 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98275b615c5949849d981d1518ee01a1 in term 0.
I20260807 07:01:58.316188 28383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deea7403c5044ee2955f22a567e78df8" candidate_uuid: "98275b615c5949849d981d1518ee01a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ff693488f35473aab7e986e1c724020" is_pre_election: true
I20260807 07:01:58.316434 28683 leader_election.cc:302] T deea7403c5044ee2955f22a567e78df8 P 98275b615c5949849d981d1518ee01a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e76949275b94888adef92c64b34d096, 98275b615c5949849d981d1518ee01a1; no voters:
I20260807 07:01:58.316664 28383 raft_consensus.cc:2454] T deea7403c5044ee2955f22a567e78df8 P 0ff693488f35473aab7e986e1c724020 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98275b615c5949849d981d1518ee01a1 in term 0.
I20260807 07:01:58.316798 28869 raft_consensus.cc:2781] T deea7403c5044ee2955f22a567e78df8 P 98275b615c5949849d981d1518ee01a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 07:01:58.316974 28869 raft_consensus.cc:482] T deea7403c5044ee2955f22a567e78df8 P 98275b615c5949849d981d1518ee01a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:01:58.317082 28869 raft_consensus.cc:3037] T deea7403c5044ee2955f22a567e78df8 P 98275b615c5949849d981d1518ee01a1 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:01:58.319208 28869 raft_consensus.cc:504] T deea7403c5044ee2955f22a567e78df8 P 98275b615c5949849d981d1518ee01a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98275b615c5949849d981d1518ee01a1" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 46765 } } peers { permanent_uuid: "0ff693488f35473aab7e986e1c724020" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 41029 } } peers { permanent_uuid: "8e76949275b94888adef92c64b34d096" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 38737 } }
I20260807 07:01:58.319772 28869 leader_election.cc:288] T deea7403c5044ee2955f22a567e78df8 P 98275b615c5949849d981d1518ee01a1 [CANDIDATE]: Term 1 election: Requested vote from peers 0ff693488f35473aab7e986e1c724020 (127.21.83.1:41029), 8e76949275b94888adef92c64b34d096 (127.21.83.2:38737)
I20260807 07:01:58.320030 28383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deea7403c5044ee2955f22a567e78df8" candidate_uuid: "98275b615c5949849d981d1518ee01a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ff693488f35473aab7e986e1c724020"
I20260807 07:01:58.320219 28383 raft_consensus.cc:3037] T deea7403c5044ee2955f22a567e78df8 P 0ff693488f35473aab7e986e1c724020 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:01:58.320214 28589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deea7403c5044ee2955f22a567e78df8" candidate_uuid: "98275b615c5949849d981d1518ee01a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e76949275b94888adef92c64b34d096"
I20260807 07:01:58.320384 28589 raft_consensus.cc:3037] T deea7403c5044ee2955f22a567e78df8 P 8e76949275b94888adef92c64b34d096 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:01:58.323366 28383 raft_consensus.cc:2454] T deea7403c5044ee2955f22a567e78df8 P 0ff693488f35473aab7e986e1c724020 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98275b615c5949849d981d1518ee01a1 in term 1.
I20260807 07:01:58.323431 28589 raft_consensus.cc:2454] T deea7403c5044ee2955f22a567e78df8 P 8e76949275b94888adef92c64b34d096 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98275b615c5949849d981d1518ee01a1 in term 1.
I20260807 07:01:58.323741 28683 leader_election.cc:302] T deea7403c5044ee2955f22a567e78df8 P 98275b615c5949849d981d1518ee01a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e76949275b94888adef92c64b34d096, 98275b615c5949849d981d1518ee01a1; no voters:
I20260807 07:01:58.324023 28869 raft_consensus.cc:2781] T deea7403c5044ee2955f22a567e78df8 P 98275b615c5949849d981d1518ee01a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:01:58.324445 28869 raft_consensus.cc:686] T deea7403c5044ee2955f22a567e78df8 P 98275b615c5949849d981d1518ee01a1 [term 1 LEADER]: Becoming Leader. State: Replica: 98275b615c5949849d981d1518ee01a1, State: Running, Role: LEADER
I20260807 07:01:58.324910 28869 consensus_queue.cc:237] T deea7403c5044ee2955f22a567e78df8 P 98275b615c5949849d981d1518ee01a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98275b615c5949849d981d1518ee01a1" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 46765 } } peers { permanent_uuid: "0ff693488f35473aab7e986e1c724020" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 41029 } } peers { permanent_uuid: "8e76949275b94888adef92c64b34d096" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 38737 } }
W20260807 07:01:58.328099 28654 tablet.cc:2369] T deea7403c5044ee2955f22a567e78df8 P 8e76949275b94888adef92c64b34d096: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 07:01:58.329228 28177 catalog_manager.cc:5495] T deea7403c5044ee2955f22a567e78df8 P 98275b615c5949849d981d1518ee01a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 98275b615c5949849d981d1518ee01a1 (127.21.83.3). New cstate: current_term: 1 leader_uuid: "98275b615c5949849d981d1518ee01a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98275b615c5949849d981d1518ee01a1" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 46765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ff693488f35473aab7e986e1c724020" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 41029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e76949275b94888adef92c64b34d096" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 38737 } health_report { overall_health: UNKNOWN } } }
I20260807 07:01:58.348454 28852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:58.343367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34318 (local address 127.21.83.3:46765)
0807 07:01:58.343533 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:58.343539 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:58.343732 (+ 193us) server_negotiation.cc:407] Connection header received
0807 07:01:58.343932 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:58.343935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:58.344004 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:58.344123 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:01:58.344495 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:58.345222 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:58.346514 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:58.346896 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:58.347000 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:58.347470 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:01:58.347918 (+ 448us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:01:58.348045 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:58.348150 (+ 105us) server_negotiation.cc:299] Negotiation successful
0807 07:01:58.348230 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260807 07:01:58.355810 28589 raft_consensus.cc:1261] T deea7403c5044ee2955f22a567e78df8 P 8e76949275b94888adef92c64b34d096 [term 1 FOLLOWER]: Refusing update from remote peer 98275b615c5949849d981d1518ee01a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:01:58.355810 28383 raft_consensus.cc:1261] T deea7403c5044ee2955f22a567e78df8 P 0ff693488f35473aab7e986e1c724020 [term 1 FOLLOWER]: Refusing update from remote peer 98275b615c5949849d981d1518ee01a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:01:58.357028 28869 consensus_queue.cc:1035] T deea7403c5044ee2955f22a567e78df8 P 98275b615c5949849d981d1518ee01a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ff693488f35473aab7e986e1c724020" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 41029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:01:58.357383 28871 consensus_queue.cc:1035] T deea7403c5044ee2955f22a567e78df8 P 98275b615c5949849d981d1518ee01a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e76949275b94888adef92c64b34d096" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:01:58.397837 28175 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:01:58.398180 28175 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:01:58.402009 28175 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b06392ed002d4c82adb03efbb0cbb62e: Sending DeleteTablet for 3 replicas of tablet deea7403c5044ee2955f22a567e78df8
I20260807 07:01:58.402994 28744 tablet_service.cc:1514] Processing DeleteTablet for tablet deea7403c5044ee2955f22a567e78df8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:01:58 UTC) from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:34312
I20260807 07:01:58.403180 28358 tablet_service.cc:1514] Processing DeleteTablet for tablet deea7403c5044ee2955f22a567e78df8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:01:58 UTC) from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:53632
I20260807 07:01:58.404229 28553 tablet_service.cc:1514] Processing DeleteTablet for tablet deea7403c5044ee2955f22a567e78df8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:01:58 UTC) from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:56762
I20260807 07:01:58.404647 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 28264
I20260807 07:01:58.413805 28892 tablet_replica.cc:330] T deea7403c5044ee2955f22a567e78df8 P 0ff693488f35473aab7e986e1c724020: stopping tablet replica
I20260807 07:01:58.414284 28892 raft_consensus.cc:2229] T deea7403c5044ee2955f22a567e78df8 P 0ff693488f35473aab7e986e1c724020 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:01:58.414794 28892 raft_consensus.cc:2258] T deea7403c5044ee2955f22a567e78df8 P 0ff693488f35473aab7e986e1c724020 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:01:58.416262 28893 tablet_replica.cc:330] T deea7403c5044ee2955f22a567e78df8 P 8e76949275b94888adef92c64b34d096: stopping tablet replica
I20260807 07:01:58.416690 28893 raft_consensus.cc:2229] T deea7403c5044ee2955f22a567e78df8 P 8e76949275b94888adef92c64b34d096 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:01:58.417210 28893 raft_consensus.cc:2258] T deea7403c5044ee2955f22a567e78df8 P 8e76949275b94888adef92c64b34d096 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:01:58.417429 28892 ts_tablet_manager.cc:1905] T deea7403c5044ee2955f22a567e78df8 P 0ff693488f35473aab7e986e1c724020: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:01:58.418401 28891 tablet_replica.cc:330] T deea7403c5044ee2955f22a567e78df8 P 98275b615c5949849d981d1518ee01a1: stopping tablet replica
I20260807 07:01:58.418795 28891 raft_consensus.cc:2229] T deea7403c5044ee2955f22a567e78df8 P 98275b615c5949849d981d1518ee01a1 [term 1 LEADER]: Raft consensus shutting down.
I20260807 07:01:58.419479 28891 raft_consensus.cc:2258] T deea7403c5044ee2955f22a567e78df8 P 98275b615c5949849d981d1518ee01a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:01:58.421339 28893 ts_tablet_manager.cc:1905] T deea7403c5044ee2955f22a567e78df8 P 8e76949275b94888adef92c64b34d096: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:01:58.425302 28892 ts_tablet_manager.cc:1918] T deea7403c5044ee2955f22a567e78df8 P 0ff693488f35473aab7e986e1c724020: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:01:58.425560 28892 log.cc:1196] T deea7403c5044ee2955f22a567e78df8 P 0ff693488f35473aab7e986e1c724020: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-0/wal/wals/deea7403c5044ee2955f22a567e78df8
I20260807 07:01:58.426200 28892 ts_tablet_manager.cc:1939] T deea7403c5044ee2955f22a567e78df8 P 0ff693488f35473aab7e986e1c724020: Deleting consensus metadata
Aug 07 07:01:58 dist-test-slave-ncp9 krb5kdc[28121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.1@KRBTEST.COM
I20260807 07:01:58.428334 28158 catalog_manager.cc:4841] TS 0ff693488f35473aab7e986e1c724020 (127.21.83.1:41029): tablet deea7403c5044ee2955f22a567e78df8 (table test-table [id=e7115d46d18b4e54b22a714d710eed1b]) successfully deleted
I20260807 07:01:58.429314 28893 ts_tablet_manager.cc:1918] T deea7403c5044ee2955f22a567e78df8 P 8e76949275b94888adef92c64b34d096: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:01:58.429513 28893 log.cc:1196] T deea7403c5044ee2955f22a567e78df8 P 8e76949275b94888adef92c64b34d096: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-1/wal/wals/deea7403c5044ee2955f22a567e78df8
I20260807 07:01:58.430063 28893 ts_tablet_manager.cc:1939] T deea7403c5044ee2955f22a567e78df8 P 8e76949275b94888adef92c64b34d096: Deleting consensus metadata
I20260807 07:01:58.431440 28891 ts_tablet_manager.cc:1905] T deea7403c5044ee2955f22a567e78df8 P 98275b615c5949849d981d1518ee01a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:01:58.431919 28157 catalog_manager.cc:4841] TS 8e76949275b94888adef92c64b34d096 (127.21.83.2:38737): tablet deea7403c5044ee2955f22a567e78df8 (table test-table [id=e7115d46d18b4e54b22a714d710eed1b]) successfully deleted
I20260807 07:01:58.436267 28854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:58.405908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53642 (local address 127.21.83.1:41029)
0807 07:01:58.406088 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:58.406094 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:58.415593 (+ 9499us) server_negotiation.cc:407] Connection header received
0807 07:01:58.415796 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:58.415801 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:58.415882 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:58.415986 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:58.419652 (+ 3666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:58.420765 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:58.422566 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:58.423257 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:58.423360 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:58.428738 (+ 5378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:58.428750 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:58.428764 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:58.428799 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:58.432858 (+ 4059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:58.433838 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:58.433848 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:58.433852 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:58.434037 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:58.434480 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:58.434485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:58.434488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:58.434981 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:58.435312 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:58.435642 (+ 330us) server_negotiation.cc:299] Negotiation successful
0807 07:01:58.435894 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 07:01:58.437038 28439 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:01:58.439069 28891 ts_tablet_manager.cc:1918] T deea7403c5044ee2955f22a567e78df8 P 98275b615c5949849d981d1518ee01a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:01:58.439286 28891 log.cc:1196] T deea7403c5044ee2955f22a567e78df8 P 98275b615c5949849d981d1518ee01a1: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786086078080656-21836-0/minicluster-data/ts-2/wal/wals/deea7403c5044ee2955f22a567e78df8
I20260807 07:01:58.440004 28891 ts_tablet_manager.cc:1939] T deea7403c5044ee2955f22a567e78df8 P 98275b615c5949849d981d1518ee01a1: Deleting consensus metadata
I20260807 07:01:58.442086 28157 catalog_manager.cc:4841] TS 98275b615c5949849d981d1518ee01a1 (127.21.83.3:46765): tablet deea7403c5044ee2955f22a567e78df8 (table test-table [id=e7115d46d18b4e54b22a714d710eed1b]) successfully deleted
I20260807 07:01:58.528302 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 28264
I20260807 07:01:58.557600 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 28454
Aug 07 07:01:58 dist-test-slave-ncp9 krb5kdc[28121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.2@KRBTEST.COM
I20260807 07:01:58.574173 28857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:58.558420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56772 (local address 127.21.83.2:38737)
0807 07:01:58.558583 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:58.558588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:58.558606 (+ 18us) server_negotiation.cc:407] Connection header received
0807 07:01:58.558730 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:58.558734 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:58.558825 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:58.558915 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:58.560454 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:58.561252 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:58.563050 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:58.563415 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:58.563501 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:58.568253 (+ 4752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:58.568267 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:58.568282 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:58.568314 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:58.570981 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:58.571729 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:58.571735 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:58.571737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:58.571857 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:58.572193 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:58.572196 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:58.572198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:58.572580 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:58.572909 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:58.573319 (+ 410us) server_negotiation.cc:299] Negotiation successful
0807 07:01:58.573531 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 07:01:58.574914 28645 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:01:58.705910 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 28454
I20260807 07:01:58.727674 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 28660
Aug 07 07:01:58 dist-test-slave-ncp9 krb5kdc[28121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.3@KRBTEST.COM
I20260807 07:01:58.744620 28852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:58.728471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40456 (local address 127.21.83.3:46765)
0807 07:01:58.728642 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:58.728647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:58.728864 (+ 217us) server_negotiation.cc:407] Connection header received
0807 07:01:58.729081 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:58.729085 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:58.729165 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:58.729262 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:58.730557 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:58.731403 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:58.732680 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:01:58.733212 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:01:58.733336 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:01:58.737949 (+ 4613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:58.737966 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:58.737988 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:01:58.738039 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:58.741533 (+ 3494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:58.742489 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:58.742497 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:58.742499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:58.742690 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:01:58.743134 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:01:58.743137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:01:58.743139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:01:58.743523 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:58.743806 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:58.744135 (+ 329us) server_negotiation.cc:299] Negotiation successful
0807 07:01:58.744322 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260807 07:01:58.745424 28828 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:01:58.867063 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 28660
I20260807 07:01:58.895669 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 28135
I20260807 07:01:58.896813 28222 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:01:59.006347 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 28135
2026-08-07T07:01:59Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4543 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-07T07:01:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:01:59Z Disabled control of system clock
I20260807 07:01:59.083431 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:39609
--webserver_interface=127.21.83.62
--webserver_port=0
--builtin_ntp_servers=127.21.83.20:39711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:39609
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260807 07:01:59.407328 28904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:01:59.407662 28904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:01:59.407790 28904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:01:59.415469 28904 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:01:59.415604 28904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:01:59.415648 28904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:01:59.416018 28904 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:01:59.416168 28904 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:01:59.426793 28904 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:39711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:39609
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:39609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:01:59.428149 28904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:01:59.430136 28904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:01:59.441924 28909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:59.441998 28910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:59.442018 28912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:01:59.443002 28904 server_base.cc:1034] running on GCE node
I20260807 07:01:59.443837 28904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:01:59.445413 28904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:01:59.446635 28904 hybrid_clock.cc:648] HybridClock initialized: now 1786086119446588 us; error 44 us; skew 500 ppm
I20260807 07:01:59.446995 28904 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 07:01:59.449757 28904 webserver.cc:460] Webserver started at http://127.21.83.62:43869/ using document root <none> and password file <none>
I20260807 07:01:59.450743 28904 fs_manager.cc:359] Metadata directory not provided
I20260807 07:01:59.450845 28904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:01:59.451138 28904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:01:59.454104 28904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "c9855fcd2561499f87e0b2300555f979"
format_stamp: "Formatted at 2026-08-07 07:01:59 on dist-test-slave-ncp9"
server_key: "92f025b1424958bb0106d9e6d73357fd"
server_key_iv: "68b15391cc8753999d0ae3fa5b5aec27"
server_key_version: "encryptionkey@0"
I20260807 07:01:59.454873 28904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "c9855fcd2561499f87e0b2300555f979"
format_stamp: "Formatted at 2026-08-07 07:01:59 on dist-test-slave-ncp9"
server_key: "92f025b1424958bb0106d9e6d73357fd"
server_key_iv: "68b15391cc8753999d0ae3fa5b5aec27"
server_key_version: "encryptionkey@0"
I20260807 07:01:59.460006 28904 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 07:01:59.463492 28920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:59.464951 28904 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 07:01:59.465144 28904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "c9855fcd2561499f87e0b2300555f979"
format_stamp: "Formatted at 2026-08-07 07:01:59 on dist-test-slave-ncp9"
server_key: "92f025b1424958bb0106d9e6d73357fd"
server_key_iv: "68b15391cc8753999d0ae3fa5b5aec27"
server_key_version: "encryptionkey@0"
I20260807 07:01:59.465376 28904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:01:59.471361 28904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:01:59.472193 28904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:01:59.472486 28904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:01:59.490289 28904 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.62:39609
I20260807 07:01:59.490331 28991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.62:39609 every 8 connection(s)
I20260807 07:01:59.491585 28904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:01:59.495198 28992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:01:59.501760 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 28904
I20260807 07:01:59.501966 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:01:59.502791 21836 external_mini_cluster.cc:1428] Setting key b8da0f9b686372912b2cf3ccfd197dd7
I20260807 07:01:59.506325 28992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9855fcd2561499f87e0b2300555f979: Bootstrap starting.
I20260807 07:01:59.509439 28992 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9855fcd2561499f87e0b2300555f979: Neither blocks nor log segments found. Creating new log.
I20260807 07:01:59.510524 28992 log.cc:824] T 00000000000000000000000000000000 P c9855fcd2561499f87e0b2300555f979: Log is configured to *not* fsync() on all Append() calls
I20260807 07:01:59.510864 28996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:01:59.504823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42180 (local address 127.21.83.62:39609)
0807 07:01:59.505788 (+ 965us) server_negotiation.cc:206] Beginning negotiation
0807 07:01:59.505799 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 07:01:59.505837 (+ 38us) server_negotiation.cc:407] Connection header received
0807 07:01:59.507484 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:01:59.507509 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:01:59.507904 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:01:59.508257 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:01:59.509312 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:01:59.509318 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:01:59.509330 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:01:59.509345 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:01:59.509411 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:01:59.509430 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:01:59.509599 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0807 07:01:59.509734 (+ 135us) server_negotiation.cc:299] Negotiation successful
0807 07:01:59.510005 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":324,"threads_started":1}
I20260807 07:01:59.513325 28992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9855fcd2561499f87e0b2300555f979: No bootstrap required, opened a new log
I20260807 07:01:59.520375 28992 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9855fcd2561499f87e0b2300555f979 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9855fcd2561499f87e0b2300555f979" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39609 } }
I20260807 07:01:59.520701 28992 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9855fcd2561499f87e0b2300555f979 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:01:59.520772 28992 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9855fcd2561499f87e0b2300555f979 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9855fcd2561499f87e0b2300555f979, State: Initialized, Role: FOLLOWER
I20260807 07:01:59.521303 28992 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9855fcd2561499f87e0b2300555f979 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9855fcd2561499f87e0b2300555f979" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39609 } }
I20260807 07:01:59.521507 28992 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9855fcd2561499f87e0b2300555f979 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:01:59.521603 28992 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9855fcd2561499f87e0b2300555f979 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:01:59.521725 28992 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9855fcd2561499f87e0b2300555f979 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:01:59.523782 28992 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9855fcd2561499f87e0b2300555f979 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9855fcd2561499f87e0b2300555f979" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39609 } }
I20260807 07:01:59.524215 28992 leader_election.cc:302] T 00000000000000000000000000000000 P c9855fcd2561499f87e0b2300555f979 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9855fcd2561499f87e0b2300555f979; no voters:
I20260807 07:01:59.524706 28992 leader_election.cc:288] T 00000000000000000000000000000000 P c9855fcd2561499f87e0b2300555f979 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 07:01:59.524883 28999 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9855fcd2561499f87e0b2300555f979 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:01:59.525626 28999 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9855fcd2561499f87e0b2300555f979 [term 1 LEADER]: Becoming Leader. State: Replica: c9855fcd2561499f87e0b2300555f979, State: Running, Role: LEADER
I20260807 07:01:59.526196 28999 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9855fcd2561499f87e0b2300555f979 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9855fcd2561499f87e0b2300555f979" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39609 } }
I20260807 07:01:59.526918 28992 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9855fcd2561499f87e0b2300555f979 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:01:59.529106 29005 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9855fcd2561499f87e0b2300555f979 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9855fcd2561499f87e0b2300555f979. Latest consensus state: current_term: 1 leader_uuid: "c9855fcd2561499f87e0b2300555f979" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9855fcd2561499f87e0b2300555f979" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39609 } } }
I20260807 07:01:59.529153 29002 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9855fcd2561499f87e0b2300555f979 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9855fcd2561499f87e0b2300555f979" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9855fcd2561499f87e0b2300555f979" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39609 } } }
I20260807 07:01:59.529486 29005 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9855fcd2561499f87e0b2300555f979 [sys.catalog]: This master's current role is: LEADER
I20260807 07:01:59.529516 29002 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9855fcd2561499f87e0b2300555f979 [sys.catalog]: This master's current role is: LEADER
I20260807 07:01:59.533124 29013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:01:59.538362 29013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:01:59.546972 29013 catalog_manager.cc:1347] Generated new cluster ID: 9a0a10564b594cdc840545e7dfc919ea
I20260807 07:01:59.547082 29013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:01:59.570670 29013 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:01:59.571123 29013 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:01:59.582084 29013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9855fcd2561499f87e0b2300555f979: Generated new TSK 0
I20260807 07:01:59.583284 29013 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 07:01:59.595831 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.1:0
--local_ip_for_outbound_sockets=127.21.83.1
--webserver_interface=127.21.83.1
--webserver_port=0
--tserver_master_addrs=127.21.83.62:39609
--builtin_ntp_servers=127.21.83.20:39711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260807 07:01:59.940480 29027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:01:59.940791 29027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:01:59.940884 29027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:01:59.948760 29027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:01:59.948879 29027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:01:59.949050 29027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.1
I20260807 07:01:59.958807 29027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:39711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.83.1
--webserver_port=0
--tserver_master_addrs=127.21.83.62:39609
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.1
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:01:59.960083 29027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:01:59.961889 29027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:01:59.974510 29033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:59.974449 29034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:01:59.974426 29037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:01:59.975374 29027 server_base.cc:1034] running on GCE node
I20260807 07:01:59.975945 29027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:01:59.976943 29027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:01:59.978156 29027 hybrid_clock.cc:648] HybridClock initialized: now 1786086119978104 us; error 41 us; skew 500 ppm
I20260807 07:01:59.978475 29027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 07:01:59.980618 29027 webserver.cc:460] Webserver started at http://127.21.83.1:38657/ using document root <none> and password file <none>
I20260807 07:01:59.981305 29027 fs_manager.cc:359] Metadata directory not provided
I20260807 07:01:59.981402 29027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:01:59.981729 29027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:01:59.984441 29027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-0/data/instance:
uuid: "cc9e4ec9d59b475cb2e1dd737d2855ae"
format_stamp: "Formatted at 2026-08-07 07:01:59 on dist-test-slave-ncp9"
server_key: "0fd5702c871ac85ccfca03279c1ac8b3"
server_key_iv: "a9d2bb971b674778e3bb8efb92174c62"
server_key_version: "encryptionkey@0"
I20260807 07:01:59.985246 29027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-0/wal/instance:
uuid: "cc9e4ec9d59b475cb2e1dd737d2855ae"
format_stamp: "Formatted at 2026-08-07 07:01:59 on dist-test-slave-ncp9"
server_key: "0fd5702c871ac85ccfca03279c1ac8b3"
server_key_iv: "a9d2bb971b674778e3bb8efb92174c62"
server_key_version: "encryptionkey@0"
I20260807 07:01:59.989996 29027 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260807 07:01:59.993185 29044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:01:59.994357 29027 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260807 07:01:59.994505 29027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-0/wal
uuid: "cc9e4ec9d59b475cb2e1dd737d2855ae"
format_stamp: "Formatted at 2026-08-07 07:01:59 on dist-test-slave-ncp9"
server_key: "0fd5702c871ac85ccfca03279c1ac8b3"
server_key_iv: "a9d2bb971b674778e3bb8efb92174c62"
server_key_version: "encryptionkey@0"
I20260807 07:01:59.994704 29027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:02:00.000118 29027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:02:00.000852 29027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:02:00.001122 29027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:02:00.001987 29027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:02:00.003329 29027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:02:00.003417 29027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:00.003531 29027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:02:00.003607 29027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:00.033339 29027 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.1:42147
I20260807 07:02:00.033389 29201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.1:42147 every 8 connection(s)
I20260807 07:02:00.034662 29027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
I20260807 07:02:00.040575 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 29027
I20260807 07:02:00.040755 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-0/wal/instance
I20260807 07:02:00.041500 21836 external_mini_cluster.cc:1428] Setting key 25ff5a06ad30e276e5e0290db630e299
I20260807 07:02:00.042280 29205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:00.037001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:57039 (local address 127.21.83.62:39609)
0807 07:02:00.037483 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:00.037487 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:00.038561 (+ 1074us) server_negotiation.cc:407] Connection header received
0807 07:02:00.040151 (+ 1590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:00.040156 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:00.040224 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:00.040342 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:00.041578 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:00.041584 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:00.041589 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:02:00.041610 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:00.041662 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:02:00.041678 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:00.041797 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:00.041954 (+ 157us) server_negotiation.cc:299] Negotiation successful
0807 07:02:00.042018 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":273,"threads_started":1}
I20260807 07:02:00.043259 29207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:00.037571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39609 (local address 127.21.83.1:57039)
0807 07:02:00.038420 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:00.038461 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:00.039913 (+ 1452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:00.040503 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:00.040510 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:00.040591 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:00.041281 (+ 690us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 07:02:00.041283 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:00.041318 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 07:02:00.041399 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:00.041770 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:00.041779 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:00.041785 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 07:02:00.041990 (+ 205us) client_negotiation.cc:240] Negotiation successful
0807 07:02:00.042213 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":245,"threads_started":1}
I20260807 07:02:00.044376 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.2:0
--local_ip_for_outbound_sockets=127.21.83.2
--webserver_interface=127.21.83.2
--webserver_port=0
--tserver_master_addrs=127.21.83.62:39609
--builtin_ntp_servers=127.21.83.20:39711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260807 07:02:00.044652 29202 heartbeater.cc:347] Connected to a master server at 127.21.83.62:39609
I20260807 07:02:00.044929 29202 heartbeater.cc:464] Registering TS with master...
I20260807 07:02:00.045476 29202 heartbeater.cc:511] Master 127.21.83.62:39609 requested a full tablet report, sending...
I20260807 07:02:00.047399 28946 ts_manager.cc:194] Registered new tserver with Master: cc9e4ec9d59b475cb2e1dd737d2855ae (127.21.83.1:42147)
W20260807 07:02:00.379472 29208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:02:00.379776 29208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:02:00.379895 29208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:02:00.387192 29208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:02:00.387284 29208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:02:00.387429 29208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.2
I20260807 07:02:00.396447 29208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:39711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.83.2
--webserver_port=0
--tserver_master_addrs=127.21.83.62:39609
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.2
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:02:00.397706 29208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:02:00.399441 29208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:02:00.411845 29214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:00.411852 29215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:00.412369 29218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:02:00.412807 29208 server_base.cc:1034] running on GCE node
I20260807 07:02:00.413411 29208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:02:00.414318 29208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:02:00.415518 29208 hybrid_clock.cc:648] HybridClock initialized: now 1786086120415462 us; error 52 us; skew 500 ppm
I20260807 07:02:00.415814 29208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 07:02:00.417969 29208 webserver.cc:460] Webserver started at http://127.21.83.2:37059/ using document root <none> and password file <none>
I20260807 07:02:00.418637 29208 fs_manager.cc:359] Metadata directory not provided
I20260807 07:02:00.418742 29208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:02:00.419062 29208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:02:00.421983 29208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-1/data/instance:
uuid: "bec810c20d114ee795a11df6257d3d1a"
format_stamp: "Formatted at 2026-08-07 07:02:00 on dist-test-slave-ncp9"
server_key: "eb3a98743d97564dfaf8215f8829622a"
server_key_iv: "182fa111314b3c92e9a39baae9d8baf5"
server_key_version: "encryptionkey@0"
I20260807 07:02:00.422845 29208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-1/wal/instance:
uuid: "bec810c20d114ee795a11df6257d3d1a"
format_stamp: "Formatted at 2026-08-07 07:02:00 on dist-test-slave-ncp9"
server_key: "eb3a98743d97564dfaf8215f8829622a"
server_key_iv: "182fa111314b3c92e9a39baae9d8baf5"
server_key_version: "encryptionkey@0"
I20260807 07:02:00.429208 29208 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260807 07:02:00.432751 29229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:00.434113 29208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 07:02:00.434278 29208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-1/wal
uuid: "bec810c20d114ee795a11df6257d3d1a"
format_stamp: "Formatted at 2026-08-07 07:02:00 on dist-test-slave-ncp9"
server_key: "eb3a98743d97564dfaf8215f8829622a"
server_key_iv: "182fa111314b3c92e9a39baae9d8baf5"
server_key_version: "encryptionkey@0"
I20260807 07:02:00.434495 29208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:02:00.440013 29208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:02:00.440811 29208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:02:00.441110 29208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:02:00.442076 29208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:02:00.443607 29208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:02:00.443717 29208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:00.443826 29208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:02:00.443897 29208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:00.475237 29208 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.2:34431
I20260807 07:02:00.475289 29389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.2:34431 every 8 connection(s)
I20260807 07:02:00.476624 29208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
I20260807 07:02:00.478284 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 29208
I20260807 07:02:00.478431 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-1/wal/instance
I20260807 07:02:00.478870 21836 external_mini_cluster.cc:1428] Setting key c110b25e17bd7c67d0d20b75a2034800
I20260807 07:02:00.481665 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.3:0
--local_ip_for_outbound_sockets=127.21.83.3
--webserver_interface=127.21.83.3
--webserver_port=0
--tserver_master_addrs=127.21.83.62:39609
--builtin_ntp_servers=127.21.83.20:39711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260807 07:02:00.485595 29205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:00.479433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:41833 (local address 127.21.83.62:39609)
0807 07:02:00.479655 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:00.479661 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:00.481569 (+ 1908us) server_negotiation.cc:407] Connection header received
0807 07:02:00.483374 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:00.483380 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:00.483449 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:00.483558 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:00.484870 (+ 1312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:00.484875 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:00.484879 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:02:00.484935 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:00.484976 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:02:00.484993 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:00.485123 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:00.485257 (+ 134us) server_negotiation.cc:299] Negotiation successful
0807 07:02:00.485324 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 07:02:00.486500 29392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:00.480051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39609 (local address 127.21.83.2:41833)
0807 07:02:00.481388 (+ 1337us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:00.481435 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:00.483063 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:00.483765 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:00.483773 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:00.483869 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:00.484546 (+ 677us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 07:02:00.484549 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:00.484598 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 07:02:00.484690 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:00.485076 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:00.485083 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:00.485088 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 07:02:00.485310 (+ 222us) client_negotiation.cc:240] Negotiation successful
0807 07:02:00.485536 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"thread_start_us":552,"threads_started":1}
I20260807 07:02:00.488055 29390 heartbeater.cc:347] Connected to a master server at 127.21.83.62:39609
I20260807 07:02:00.488382 29390 heartbeater.cc:464] Registering TS with master...
I20260807 07:02:00.489037 29390 heartbeater.cc:511] Master 127.21.83.62:39609 requested a full tablet report, sending...
I20260807 07:02:00.490705 28946 ts_manager.cc:194] Registered new tserver with Master: bec810c20d114ee795a11df6257d3d1a (127.21.83.2:34431)
W20260807 07:02:00.820482 29394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:02:00.820760 29394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:02:00.820876 29394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:02:00.828488 29394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:02:00.828590 29394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:02:00.828732 29394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.3
I20260807 07:02:00.837832 29394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:39711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.83.3
--webserver_port=0
--tserver_master_addrs=127.21.83.62:39609
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.3
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:02:00.839064 29394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:02:00.840759 29394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:02:00.853029 29404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:00.853251 29403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:00.853050 29406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:02:00.854161 29394 server_base.cc:1034] running on GCE node
I20260807 07:02:00.854712 29394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:02:00.855559 29394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:02:00.856781 29394 hybrid_clock.cc:648] HybridClock initialized: now 1786086120856724 us; error 42 us; skew 500 ppm
I20260807 07:02:00.857146 29394 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 07:02:00.859591 29394 webserver.cc:460] Webserver started at http://127.21.83.3:41251/ using document root <none> and password file <none>
I20260807 07:02:00.860357 29394 fs_manager.cc:359] Metadata directory not provided
I20260807 07:02:00.860471 29394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:02:00.860824 29394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:02:00.864070 29394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-2/data/instance:
uuid: "6365d0bf72014d59af4e51b12325a792"
format_stamp: "Formatted at 2026-08-07 07:02:00 on dist-test-slave-ncp9"
server_key: "5e55338d493e50eb8eb11b7aae83e5a8"
server_key_iv: "274d84a585eaf8a7cffad337f4d42ac5"
server_key_version: "encryptionkey@0"
I20260807 07:02:00.865070 29394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-2/wal/instance:
uuid: "6365d0bf72014d59af4e51b12325a792"
format_stamp: "Formatted at 2026-08-07 07:02:00 on dist-test-slave-ncp9"
server_key: "5e55338d493e50eb8eb11b7aae83e5a8"
server_key_iv: "274d84a585eaf8a7cffad337f4d42ac5"
server_key_version: "encryptionkey@0"
I20260807 07:02:00.871287 29394 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260807 07:02:00.875351 29413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:00.876886 29394 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 07:02:00.877126 29394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-2/wal
uuid: "6365d0bf72014d59af4e51b12325a792"
format_stamp: "Formatted at 2026-08-07 07:02:00 on dist-test-slave-ncp9"
server_key: "5e55338d493e50eb8eb11b7aae83e5a8"
server_key_iv: "274d84a585eaf8a7cffad337f4d42ac5"
server_key_version: "encryptionkey@0"
I20260807 07:02:00.877377 29394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:02:00.883623 29394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:02:00.884544 29394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:02:00.884835 29394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:02:00.885902 29394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:02:00.887449 29394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:02:00.887550 29394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:00.887668 29394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:02:00.887754 29394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:00.919872 29394 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.3:45109
I20260807 07:02:00.919929 29579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.3:45109 every 8 connection(s)
I20260807 07:02:00.921185 29394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
I20260807 07:02:00.927831 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 29394
I20260807 07:02:00.928020 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-2/wal/instance
I20260807 07:02:00.928591 21836 external_mini_cluster.cc:1428] Setting key 747f19a763147ac1a49b315084a9cf82
I20260807 07:02:00.929634 29205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:00.923797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:59173 (local address 127.21.83.62:39609)
0807 07:02:00.924014 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:00.924020 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:00.925445 (+ 1425us) server_negotiation.cc:407] Connection header received
0807 07:02:00.927146 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:00.927154 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:00.927227 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:00.927316 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:00.928858 (+ 1542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:00.928863 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:00.928866 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:02:00.928883 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:00.928947 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:02:00.928963 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:00.929071 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:00.929300 (+ 229us) server_negotiation.cc:299] Negotiation successful
0807 07:02:00.929373 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260807 07:02:00.930652 29583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:00.924268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39609 (local address 127.21.83.3:59173)
0807 07:02:00.925291 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:00.925343 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:00.926864 (+ 1521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:00.927612 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:00.927624 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:00.927753 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:00.928508 (+ 755us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 07:02:00.928512 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:00.928562 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 07:02:00.928679 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:00.929058 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:00.929070 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:00.929077 (+ 7us) client_negotiation.cc:769] Sending connection context
0807 07:02:00.929326 (+ 249us) client_negotiation.cc:240] Negotiation successful
0807 07:02:00.929580 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":280,"threads_started":1}
I20260807 07:02:00.931978 29581 heartbeater.cc:347] Connected to a master server at 127.21.83.62:39609
I20260807 07:02:00.932256 29581 heartbeater.cc:464] Registering TS with master...
I20260807 07:02:00.932843 29581 heartbeater.cc:511] Master 127.21.83.62:39609 requested a full tablet report, sending...
I20260807 07:02:00.934279 28945 ts_manager.cc:194] Registered new tserver with Master: 6365d0bf72014d59af4e51b12325a792 (127.21.83.3:45109)
I20260807 07:02:00.944367 21836 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 07:02:00.951900 29205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:00.949035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42186 (local address 127.21.83.62:39609)
0807 07:02:00.949197 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:00.949202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:00.949714 (+ 512us) server_negotiation.cc:407] Connection header received
0807 07:02:00.949890 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:00.949895 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:00.949950 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:00.950032 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:00.950961 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:00.950967 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:00.950972 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:02:00.950989 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:00.951023 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:02:00.951035 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:00.951151 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:00.951256 (+ 105us) server_negotiation.cc:299] Negotiation successful
0807 07:02:00.951322 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260807 07:02:00.957141 28945 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42186:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 07:02:00.964056 28945 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 07:02:00.981346 29598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:00.976288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.3:45109 (local address 127.0.0.1:48048)
0807 07:02:00.977867 (+ 1579us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:00.977923 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:00.978121 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:00.979163 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:00.979170 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:00.979268 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:00.980119 (+ 851us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 07:02:00.980123 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:00.980146 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 07:02:00.980223 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:00.980640 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:00.980652 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:00.980660 (+ 8us) client_negotiation.cc:769] Sending connection context
0807 07:02:00.980816 (+ 156us) client_negotiation.cc:240] Negotiation successful
0807 07:02:00.980928 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1306,"thread_start_us":1474,"threads_started":1}
I20260807 07:02:00.981328 29600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:00.976594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48048 (local address 127.21.83.3:45109)
0807 07:02:00.977463 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:00.977472 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:00.978025 (+ 553us) server_negotiation.cc:407] Connection header received
0807 07:02:00.978324 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:00.978335 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:00.978566 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:00.978732 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:00.980432 (+ 1700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:00.980442 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:00.980456 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:02:00.980476 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:00.980534 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:02:00.980550 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:00.980719 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:00.980792 (+ 73us) server_negotiation.cc:299] Negotiation successful
0807 07:02:00.980925 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":490,"threads_started":1}
I20260807 07:02:00.982003 29597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:00.976483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.2:34431 (local address 127.0.0.1:52004)
0807 07:02:00.977932 (+ 1449us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:00.977955 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:00.978079 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:00.979765 (+ 1686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:00.979770 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:00.979793 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:00.980339 (+ 546us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 07:02:00.980342 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:00.980357 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 07:02:00.980396 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:00.981127 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:00.981136 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:00.981142 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 07:02:00.981433 (+ 291us) client_negotiation.cc:240] Negotiation successful
0807 07:02:00.981502 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1305,"thread_start_us":947,"threads_started":1}
I20260807 07:02:00.982888 29602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:00.977130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52004 (local address 127.21.83.2:34431)
0807 07:02:00.978720 (+ 1590us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:00.978729 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:00.978754 (+ 25us) server_negotiation.cc:407] Connection header received
0807 07:02:00.978837 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:00.978844 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:00.979043 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:00.979193 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:00.980769 (+ 1576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:00.980777 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:00.980789 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:02:00.980812 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:00.980861 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:02:00.980878 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:00.981065 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:00.982397 (+ 1332us) server_negotiation.cc:299] Negotiation successful
0807 07:02:00.982519 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1430,"thread_start_us":1284,"threads_started":1}
I20260807 07:02:00.984248 29598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:00.978760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:42147 (local address 127.0.0.1:57172)
0807 07:02:00.981719 (+ 2959us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:00.981740 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:00.981829 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:00.982355 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:00.982359 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:00.982401 (+ 42us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:00.982886 (+ 485us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 07:02:00.982888 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:00.982901 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 07:02:00.982935 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:00.983792 (+ 857us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:00.983797 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:00.983802 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 07:02:00.983894 (+ 92us) client_negotiation.cc:240] Negotiation successful
0807 07:02:00.983942 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2860,"threads_started":1}
I20260807 07:02:00.984297 29603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:00.978514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57172 (local address 127.21.83.1:42147)
0807 07:02:00.979024 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:00.979032 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:00.981936 (+ 2904us) server_negotiation.cc:407] Connection header received
0807 07:02:00.981997 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:00.982001 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:00.982166 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:00.982679 (+ 513us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:00.983612 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:00.983617 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:00.983624 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:02:00.983635 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:00.983669 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:02:00.983685 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:00.983804 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:00.983909 (+ 105us) server_negotiation.cc:299] Negotiation successful
0807 07:02:00.984005 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":349,"threads_started":1}
I20260807 07:02:00.986649 29299 tablet_service.cc:1467] Processing CreateTablet for tablet 953f2928eba84e898c8d29e5aa8f1b76 (DEFAULT_TABLE table=test-table [id=ff08a46831cc4dc286318c8a0d99c48d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:02:00.987867 29299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 953f2928eba84e898c8d29e5aa8f1b76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:02:00.987898 29114 tablet_service.cc:1467] Processing CreateTablet for tablet 953f2928eba84e898c8d29e5aa8f1b76 (DEFAULT_TABLE table=test-table [id=ff08a46831cc4dc286318c8a0d99c48d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:02:00.987897 29481 tablet_service.cc:1467] Processing CreateTablet for tablet 953f2928eba84e898c8d29e5aa8f1b76 (DEFAULT_TABLE table=test-table [id=ff08a46831cc4dc286318c8a0d99c48d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:02:00.989195 29481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 953f2928eba84e898c8d29e5aa8f1b76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:02:00.989205 29114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 953f2928eba84e898c8d29e5aa8f1b76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:02:00.999380 29605 tablet_bootstrap.cc:492] T 953f2928eba84e898c8d29e5aa8f1b76 P bec810c20d114ee795a11df6257d3d1a: Bootstrap starting.
I20260807 07:02:01.002799 29605 tablet_bootstrap.cc:654] T 953f2928eba84e898c8d29e5aa8f1b76 P bec810c20d114ee795a11df6257d3d1a: Neither blocks nor log segments found. Creating new log.
I20260807 07:02:01.003633 29607 tablet_bootstrap.cc:492] T 953f2928eba84e898c8d29e5aa8f1b76 P cc9e4ec9d59b475cb2e1dd737d2855ae: Bootstrap starting.
I20260807 07:02:01.004034 29605 log.cc:824] T 953f2928eba84e898c8d29e5aa8f1b76 P bec810c20d114ee795a11df6257d3d1a: Log is configured to *not* fsync() on all Append() calls
I20260807 07:02:01.008147 29607 tablet_bootstrap.cc:654] T 953f2928eba84e898c8d29e5aa8f1b76 P cc9e4ec9d59b475cb2e1dd737d2855ae: Neither blocks nor log segments found. Creating new log.
I20260807 07:02:01.009034 29609 tablet_bootstrap.cc:492] T 953f2928eba84e898c8d29e5aa8f1b76 P 6365d0bf72014d59af4e51b12325a792: Bootstrap starting.
I20260807 07:02:01.009513 29607 log.cc:824] T 953f2928eba84e898c8d29e5aa8f1b76 P cc9e4ec9d59b475cb2e1dd737d2855ae: Log is configured to *not* fsync() on all Append() calls
I20260807 07:02:01.012709 29605 tablet_bootstrap.cc:492] T 953f2928eba84e898c8d29e5aa8f1b76 P bec810c20d114ee795a11df6257d3d1a: No bootstrap required, opened a new log
I20260807 07:02:01.012676 29609 tablet_bootstrap.cc:654] T 953f2928eba84e898c8d29e5aa8f1b76 P 6365d0bf72014d59af4e51b12325a792: Neither blocks nor log segments found. Creating new log.
I20260807 07:02:01.012995 29605 ts_tablet_manager.cc:1397] T 953f2928eba84e898c8d29e5aa8f1b76 P bec810c20d114ee795a11df6257d3d1a: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.008s
I20260807 07:02:01.013126 29607 tablet_bootstrap.cc:492] T 953f2928eba84e898c8d29e5aa8f1b76 P cc9e4ec9d59b475cb2e1dd737d2855ae: No bootstrap required, opened a new log
I20260807 07:02:01.013399 29607 ts_tablet_manager.cc:1397] T 953f2928eba84e898c8d29e5aa8f1b76 P cc9e4ec9d59b475cb2e1dd737d2855ae: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260807 07:02:01.013954 29609 log.cc:824] T 953f2928eba84e898c8d29e5aa8f1b76 P 6365d0bf72014d59af4e51b12325a792: Log is configured to *not* fsync() on all Append() calls
I20260807 07:02:01.017298 29609 tablet_bootstrap.cc:492] T 953f2928eba84e898c8d29e5aa8f1b76 P 6365d0bf72014d59af4e51b12325a792: No bootstrap required, opened a new log
I20260807 07:02:01.017560 29609 ts_tablet_manager.cc:1397] T 953f2928eba84e898c8d29e5aa8f1b76 P 6365d0bf72014d59af4e51b12325a792: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260807 07:02:01.020527 29605 raft_consensus.cc:348] T 953f2928eba84e898c8d29e5aa8f1b76 P bec810c20d114ee795a11df6257d3d1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6365d0bf72014d59af4e51b12325a792" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 45109 } } peers { permanent_uuid: "bec810c20d114ee795a11df6257d3d1a" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 34431 } } peers { permanent_uuid: "cc9e4ec9d59b475cb2e1dd737d2855ae" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42147 } }
I20260807 07:02:01.020864 29605 raft_consensus.cc:374] T 953f2928eba84e898c8d29e5aa8f1b76 P bec810c20d114ee795a11df6257d3d1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:02:01.021011 29605 raft_consensus.cc:729] T 953f2928eba84e898c8d29e5aa8f1b76 P bec810c20d114ee795a11df6257d3d1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bec810c20d114ee795a11df6257d3d1a, State: Initialized, Role: FOLLOWER
I20260807 07:02:01.021687 29605 consensus_queue.cc:260] T 953f2928eba84e898c8d29e5aa8f1b76 P bec810c20d114ee795a11df6257d3d1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6365d0bf72014d59af4e51b12325a792" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 45109 } } peers { permanent_uuid: "bec810c20d114ee795a11df6257d3d1a" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 34431 } } peers { permanent_uuid: "cc9e4ec9d59b475cb2e1dd737d2855ae" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42147 } }
I20260807 07:02:01.022882 29390 heartbeater.cc:503] Master 127.21.83.62:39609 was elected leader, sending a full tablet report...
I20260807 07:02:01.024379 29605 ts_tablet_manager.cc:1428] T 953f2928eba84e898c8d29e5aa8f1b76 P bec810c20d114ee795a11df6257d3d1a: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
I20260807 07:02:01.024400 29607 raft_consensus.cc:348] T 953f2928eba84e898c8d29e5aa8f1b76 P cc9e4ec9d59b475cb2e1dd737d2855ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6365d0bf72014d59af4e51b12325a792" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 45109 } } peers { permanent_uuid: "bec810c20d114ee795a11df6257d3d1a" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 34431 } } peers { permanent_uuid: "cc9e4ec9d59b475cb2e1dd737d2855ae" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42147 } }
I20260807 07:02:01.024838 29607 raft_consensus.cc:374] T 953f2928eba84e898c8d29e5aa8f1b76 P cc9e4ec9d59b475cb2e1dd737d2855ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:02:01.024950 29607 raft_consensus.cc:729] T 953f2928eba84e898c8d29e5aa8f1b76 P cc9e4ec9d59b475cb2e1dd737d2855ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc9e4ec9d59b475cb2e1dd737d2855ae, State: Initialized, Role: FOLLOWER
I20260807 07:02:01.025543 29607 consensus_queue.cc:260] T 953f2928eba84e898c8d29e5aa8f1b76 P cc9e4ec9d59b475cb2e1dd737d2855ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6365d0bf72014d59af4e51b12325a792" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 45109 } } peers { permanent_uuid: "bec810c20d114ee795a11df6257d3d1a" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 34431 } } peers { permanent_uuid: "cc9e4ec9d59b475cb2e1dd737d2855ae" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42147 } }
I20260807 07:02:01.026620 29202 heartbeater.cc:503] Master 127.21.83.62:39609 was elected leader, sending a full tablet report...
I20260807 07:02:01.027309 29607 ts_tablet_manager.cc:1428] T 953f2928eba84e898c8d29e5aa8f1b76 P cc9e4ec9d59b475cb2e1dd737d2855ae: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260807 07:02:01.027504 29609 raft_consensus.cc:348] T 953f2928eba84e898c8d29e5aa8f1b76 P 6365d0bf72014d59af4e51b12325a792 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6365d0bf72014d59af4e51b12325a792" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 45109 } } peers { permanent_uuid: "bec810c20d114ee795a11df6257d3d1a" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 34431 } } peers { permanent_uuid: "cc9e4ec9d59b475cb2e1dd737d2855ae" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42147 } }
I20260807 07:02:01.027938 29609 raft_consensus.cc:374] T 953f2928eba84e898c8d29e5aa8f1b76 P 6365d0bf72014d59af4e51b12325a792 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:02:01.028057 29609 raft_consensus.cc:729] T 953f2928eba84e898c8d29e5aa8f1b76 P 6365d0bf72014d59af4e51b12325a792 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6365d0bf72014d59af4e51b12325a792, State: Initialized, Role: FOLLOWER
I20260807 07:02:01.028587 29609 consensus_queue.cc:260] T 953f2928eba84e898c8d29e5aa8f1b76 P 6365d0bf72014d59af4e51b12325a792 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6365d0bf72014d59af4e51b12325a792" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 45109 } } peers { permanent_uuid: "bec810c20d114ee795a11df6257d3d1a" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 34431 } } peers { permanent_uuid: "cc9e4ec9d59b475cb2e1dd737d2855ae" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42147 } }
I20260807 07:02:01.029637 29581 heartbeater.cc:503] Master 127.21.83.62:39609 was elected leader, sending a full tablet report...
I20260807 07:02:01.030278 29609 ts_tablet_manager.cc:1428] T 953f2928eba84e898c8d29e5aa8f1b76 P 6365d0bf72014d59af4e51b12325a792: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
W20260807 07:02:01.036742 29203 tablet.cc:2369] T 953f2928eba84e898c8d29e5aa8f1b76 P cc9e4ec9d59b475cb2e1dd737d2855ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 07:02:01.047847 29617 raft_consensus.cc:482] T 953f2928eba84e898c8d29e5aa8f1b76 P 6365d0bf72014d59af4e51b12325a792 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:02:01.048071 29617 raft_consensus.cc:504] T 953f2928eba84e898c8d29e5aa8f1b76 P 6365d0bf72014d59af4e51b12325a792 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6365d0bf72014d59af4e51b12325a792" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 45109 } } peers { permanent_uuid: "bec810c20d114ee795a11df6257d3d1a" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 34431 } } peers { permanent_uuid: "cc9e4ec9d59b475cb2e1dd737d2855ae" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42147 } }
I20260807 07:02:01.049546 29617 leader_election.cc:288] T 953f2928eba84e898c8d29e5aa8f1b76 P 6365d0bf72014d59af4e51b12325a792 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bec810c20d114ee795a11df6257d3d1a (127.21.83.2:34431), cc9e4ec9d59b475cb2e1dd737d2855ae (127.21.83.1:42147)
I20260807 07:02:01.052202 29603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:01.050052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:41169 (local address 127.21.83.1:42147)
0807 07:02:01.050242 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:01.050248 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:01.050353 (+ 105us) server_negotiation.cc:407] Connection header received
0807 07:02:01.050515 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:01.050520 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:01.050583 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:01.050681 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:01.051597 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:01.051603 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:01.051608 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:02:01.051628 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:01.051663 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:02:01.051675 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:01.051774 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:01.051901 (+ 127us) server_negotiation.cc:299] Negotiation successful
0807 07:02:01.051959 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 07:02:01.052212 29583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:01.049935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:42147 (local address 127.21.83.3:41169)
0807 07:02:01.050240 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:01.050260 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:01.050371 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:01.050734 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:01.050740 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:01.050763 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:01.051355 (+ 592us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 07:02:01.051358 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:01.051378 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 07:02:01.051426 (+ 48us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:01.051780 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:01.051785 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:01.051791 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 07:02:01.051884 (+ 93us) client_negotiation.cc:240] Negotiation successful
0807 07:02:01.051940 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181}
I20260807 07:02:01.052362 29602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:01.050046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:44341 (local address 127.21.83.2:34431)
0807 07:02:01.050285 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:01.050291 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:01.050802 (+ 511us) server_negotiation.cc:407] Connection header received
0807 07:02:01.050916 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:01.050920 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:01.050967 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:01.051059 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:01.051771 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:01.051777 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:01.051780 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:02:01.051795 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:01.051825 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:02:01.051836 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:01.051940 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:01.052050 (+ 110us) server_negotiation.cc:299] Negotiation successful
0807 07:02:01.052108 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260807 07:02:01.052554 29618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:01.049936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.2:34431 (local address 127.21.83.3:44341)
0807 07:02:01.050706 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:01.050721 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:01.050821 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:01.051086 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:01.051089 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:01.051108 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:01.051599 (+ 491us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 07:02:01.051601 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:01.051613 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 07:02:01.051642 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:01.051916 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:01.051921 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:01.051926 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 07:02:01.052045 (+ 119us) client_negotiation.cc:240] Negotiation successful
0807 07:02:01.052097 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"mutex_wait_us":46,"thread_start_us":400,"threads_started":1}
I20260807 07:02:01.053371 29138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953f2928eba84e898c8d29e5aa8f1b76" candidate_uuid: "6365d0bf72014d59af4e51b12325a792" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc9e4ec9d59b475cb2e1dd737d2855ae" is_pre_election: true
I20260807 07:02:01.053545 29323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953f2928eba84e898c8d29e5aa8f1b76" candidate_uuid: "6365d0bf72014d59af4e51b12325a792" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bec810c20d114ee795a11df6257d3d1a" is_pre_election: true
I20260807 07:02:01.053812 29138 raft_consensus.cc:2454] T 953f2928eba84e898c8d29e5aa8f1b76 P cc9e4ec9d59b475cb2e1dd737d2855ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6365d0bf72014d59af4e51b12325a792 in term 0.
I20260807 07:02:01.054016 29323 raft_consensus.cc:2454] T 953f2928eba84e898c8d29e5aa8f1b76 P bec810c20d114ee795a11df6257d3d1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6365d0bf72014d59af4e51b12325a792 in term 0.
I20260807 07:02:01.054595 29422 leader_election.cc:302] T 953f2928eba84e898c8d29e5aa8f1b76 P 6365d0bf72014d59af4e51b12325a792 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6365d0bf72014d59af4e51b12325a792, bec810c20d114ee795a11df6257d3d1a, cc9e4ec9d59b475cb2e1dd737d2855ae; no voters:
I20260807 07:02:01.054994 29617 raft_consensus.cc:2781] T 953f2928eba84e898c8d29e5aa8f1b76 P 6365d0bf72014d59af4e51b12325a792 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 07:02:01.055127 29617 raft_consensus.cc:482] T 953f2928eba84e898c8d29e5aa8f1b76 P 6365d0bf72014d59af4e51b12325a792 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:02:01.055222 29617 raft_consensus.cc:3037] T 953f2928eba84e898c8d29e5aa8f1b76 P 6365d0bf72014d59af4e51b12325a792 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:02:01.057430 29617 raft_consensus.cc:504] T 953f2928eba84e898c8d29e5aa8f1b76 P 6365d0bf72014d59af4e51b12325a792 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6365d0bf72014d59af4e51b12325a792" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 45109 } } peers { permanent_uuid: "bec810c20d114ee795a11df6257d3d1a" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 34431 } } peers { permanent_uuid: "cc9e4ec9d59b475cb2e1dd737d2855ae" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42147 } }
I20260807 07:02:01.057931 29617 leader_election.cc:288] T 953f2928eba84e898c8d29e5aa8f1b76 P 6365d0bf72014d59af4e51b12325a792 [CANDIDATE]: Term 1 election: Requested vote from peers bec810c20d114ee795a11df6257d3d1a (127.21.83.2:34431), cc9e4ec9d59b475cb2e1dd737d2855ae (127.21.83.1:42147)
I20260807 07:02:01.058336 29323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953f2928eba84e898c8d29e5aa8f1b76" candidate_uuid: "6365d0bf72014d59af4e51b12325a792" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bec810c20d114ee795a11df6257d3d1a"
I20260807 07:02:01.058352 29138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953f2928eba84e898c8d29e5aa8f1b76" candidate_uuid: "6365d0bf72014d59af4e51b12325a792" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc9e4ec9d59b475cb2e1dd737d2855ae"
I20260807 07:02:01.058485 29323 raft_consensus.cc:3037] T 953f2928eba84e898c8d29e5aa8f1b76 P bec810c20d114ee795a11df6257d3d1a [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:02:01.058511 29138 raft_consensus.cc:3037] T 953f2928eba84e898c8d29e5aa8f1b76 P cc9e4ec9d59b475cb2e1dd737d2855ae [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:02:01.060555 29323 raft_consensus.cc:2454] T 953f2928eba84e898c8d29e5aa8f1b76 P bec810c20d114ee795a11df6257d3d1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6365d0bf72014d59af4e51b12325a792 in term 1.
I20260807 07:02:01.060644 29138 raft_consensus.cc:2454] T 953f2928eba84e898c8d29e5aa8f1b76 P cc9e4ec9d59b475cb2e1dd737d2855ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6365d0bf72014d59af4e51b12325a792 in term 1.
I20260807 07:02:01.060949 29418 leader_election.cc:302] T 953f2928eba84e898c8d29e5aa8f1b76 P 6365d0bf72014d59af4e51b12325a792 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6365d0bf72014d59af4e51b12325a792, bec810c20d114ee795a11df6257d3d1a; no voters:
I20260807 07:02:01.061276 29617 raft_consensus.cc:2781] T 953f2928eba84e898c8d29e5aa8f1b76 P 6365d0bf72014d59af4e51b12325a792 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:02:01.061687 29617 raft_consensus.cc:686] T 953f2928eba84e898c8d29e5aa8f1b76 P 6365d0bf72014d59af4e51b12325a792 [term 1 LEADER]: Becoming Leader. State: Replica: 6365d0bf72014d59af4e51b12325a792, State: Running, Role: LEADER
I20260807 07:02:01.062083 29617 consensus_queue.cc:237] T 953f2928eba84e898c8d29e5aa8f1b76 P 6365d0bf72014d59af4e51b12325a792 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6365d0bf72014d59af4e51b12325a792" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 45109 } } peers { permanent_uuid: "bec810c20d114ee795a11df6257d3d1a" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 34431 } } peers { permanent_uuid: "cc9e4ec9d59b475cb2e1dd737d2855ae" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42147 } }
I20260807 07:02:01.066218 28946 catalog_manager.cc:5495] T 953f2928eba84e898c8d29e5aa8f1b76 P 6365d0bf72014d59af4e51b12325a792 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6365d0bf72014d59af4e51b12325a792 (127.21.83.3). New cstate: current_term: 1 leader_uuid: "6365d0bf72014d59af4e51b12325a792" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6365d0bf72014d59af4e51b12325a792" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 45109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bec810c20d114ee795a11df6257d3d1a" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 34431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc9e4ec9d59b475cb2e1dd737d2855ae" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42147 } health_report { overall_health: UNKNOWN } } }
I20260807 07:02:01.092527 29600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:01.090408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48064 (local address 127.21.83.3:45109)
0807 07:02:01.090578 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:01.090583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:01.090822 (+ 239us) server_negotiation.cc:407] Connection header received
0807 07:02:01.091004 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:01.091009 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:01.091101 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:01.091191 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:01.092061 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:01.092065 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:01.092068 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:02:01.092085 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:01.092118 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:02:01.092129 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:01.092235 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:01.092272 (+ 37us) server_negotiation.cc:299] Negotiation successful
0807 07:02:01.092320 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 07:02:01.101284 29138 raft_consensus.cc:1261] T 953f2928eba84e898c8d29e5aa8f1b76 P cc9e4ec9d59b475cb2e1dd737d2855ae [term 1 FOLLOWER]: Refusing update from remote peer 6365d0bf72014d59af4e51b12325a792: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:02:01.101284 29323 raft_consensus.cc:1261] T 953f2928eba84e898c8d29e5aa8f1b76 P bec810c20d114ee795a11df6257d3d1a [term 1 FOLLOWER]: Refusing update from remote peer 6365d0bf72014d59af4e51b12325a792: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:02:01.102283 29619 consensus_queue.cc:1035] T 953f2928eba84e898c8d29e5aa8f1b76 P 6365d0bf72014d59af4e51b12325a792 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc9e4ec9d59b475cb2e1dd737d2855ae" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:02:01.102685 29617 consensus_queue.cc:1035] T 953f2928eba84e898c8d29e5aa8f1b76 P 6365d0bf72014d59af4e51b12325a792 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bec810c20d114ee795a11df6257d3d1a" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 34431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:02:01.114624 29625 mvcc.cc:205] Tried to move back new op lower bound from 7315808752019902464 to 7315808751874146304. Current Snapshot: MvccSnapshot[applied={T|T < 7315808752019902464}]
I20260807 07:02:01.115067 29622 mvcc.cc:205] Tried to move back new op lower bound from 7315808752019902464 to 7315808751874146304. Current Snapshot: MvccSnapshot[applied={T|T < 7315808752019902464}]
I20260807 07:02:01.127409 28945 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:02:01.127715 28945 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:02:01.131460 28945 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c9855fcd2561499f87e0b2300555f979: Sending DeleteTablet for 3 replicas of tablet 953f2928eba84e898c8d29e5aa8f1b76
I20260807 07:02:01.132608 29299 tablet_service.cc:1514] Processing DeleteTablet for tablet 953f2928eba84e898c8d29e5aa8f1b76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:02:01 UTC) from {username='slave'} at 127.0.0.1:52004
I20260807 07:02:01.133028 29481 tablet_service.cc:1514] Processing DeleteTablet for tablet 953f2928eba84e898c8d29e5aa8f1b76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:02:01 UTC) from {username='slave'} at 127.0.0.1:48048
I20260807 07:02:01.133705 29114 tablet_service.cc:1514] Processing DeleteTablet for tablet 953f2928eba84e898c8d29e5aa8f1b76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:02:01 UTC) from {username='slave'} at 127.0.0.1:57172
I20260807 07:02:01.133865 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 29027
I20260807 07:02:01.142117 29639 tablet_replica.cc:330] T 953f2928eba84e898c8d29e5aa8f1b76 P cc9e4ec9d59b475cb2e1dd737d2855ae: stopping tablet replica
I20260807 07:02:01.142648 29639 raft_consensus.cc:2229] T 953f2928eba84e898c8d29e5aa8f1b76 P cc9e4ec9d59b475cb2e1dd737d2855ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:02:01.143234 29639 raft_consensus.cc:2258] T 953f2928eba84e898c8d29e5aa8f1b76 P cc9e4ec9d59b475cb2e1dd737d2855ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:02:01.144114 29637 tablet_replica.cc:330] T 953f2928eba84e898c8d29e5aa8f1b76 P 6365d0bf72014d59af4e51b12325a792: stopping tablet replica
I20260807 07:02:01.144258 29603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:01.135118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57186 (local address 127.21.83.1:42147)
0807 07:02:01.135301 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:01.135307 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:01.141425 (+ 6118us) server_negotiation.cc:407] Connection header received
0807 07:02:01.141615 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:01.141620 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:01.141755 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:01.141871 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:01.143323 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:01.143328 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:01.143333 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:02:01.143354 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:01.143399 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:02:01.143417 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:01.143514 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:01.143850 (+ 336us) server_negotiation.cc:299] Negotiation successful
0807 07:02:01.143920 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 07:02:01.144541 29637 raft_consensus.cc:2229] T 953f2928eba84e898c8d29e5aa8f1b76 P 6365d0bf72014d59af4e51b12325a792 [term 1 LEADER]: Raft consensus shutting down.
I20260807 07:02:01.145386 29636 tablet_replica.cc:330] T 953f2928eba84e898c8d29e5aa8f1b76 P bec810c20d114ee795a11df6257d3d1a: stopping tablet replica
I20260807 07:02:01.145534 29196 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:02:01.145823 29636 raft_consensus.cc:2229] T 953f2928eba84e898c8d29e5aa8f1b76 P bec810c20d114ee795a11df6257d3d1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:02:01.146342 29636 raft_consensus.cc:2258] T 953f2928eba84e898c8d29e5aa8f1b76 P bec810c20d114ee795a11df6257d3d1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:02:01.146797 29637 raft_consensus.cc:2258] T 953f2928eba84e898c8d29e5aa8f1b76 P 6365d0bf72014d59af4e51b12325a792 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:02:01.148655 29636 ts_tablet_manager.cc:1905] T 953f2928eba84e898c8d29e5aa8f1b76 P bec810c20d114ee795a11df6257d3d1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:02:01.149099 29637 ts_tablet_manager.cc:1905] T 953f2928eba84e898c8d29e5aa8f1b76 P 6365d0bf72014d59af4e51b12325a792: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:02:01.156857 29637 ts_tablet_manager.cc:1918] T 953f2928eba84e898c8d29e5aa8f1b76 P 6365d0bf72014d59af4e51b12325a792: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:02:01.157202 29637 log.cc:1196] T 953f2928eba84e898c8d29e5aa8f1b76 P 6365d0bf72014d59af4e51b12325a792: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-2/wal/wals/953f2928eba84e898c8d29e5aa8f1b76
I20260807 07:02:01.158838 29636 ts_tablet_manager.cc:1918] T 953f2928eba84e898c8d29e5aa8f1b76 P bec810c20d114ee795a11df6257d3d1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:02:01.159224 29636 log.cc:1196] T 953f2928eba84e898c8d29e5aa8f1b76 P bec810c20d114ee795a11df6257d3d1a: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786086078080656-21836-0/minicluster-data/ts-1/wal/wals/953f2928eba84e898c8d29e5aa8f1b76
I20260807 07:02:01.159983 29636 ts_tablet_manager.cc:1939] T 953f2928eba84e898c8d29e5aa8f1b76 P bec810c20d114ee795a11df6257d3d1a: Deleting consensus metadata
I20260807 07:02:01.159979 29637 ts_tablet_manager.cc:1939] T 953f2928eba84e898c8d29e5aa8f1b76 P 6365d0bf72014d59af4e51b12325a792: Deleting consensus metadata
I20260807 07:02:01.162022 28926 catalog_manager.cc:4841] TS 6365d0bf72014d59af4e51b12325a792 (127.21.83.3:45109): tablet 953f2928eba84e898c8d29e5aa8f1b76 (table test-table [id=ff08a46831cc4dc286318c8a0d99c48d]) successfully deleted
I20260807 07:02:01.162050 28924 catalog_manager.cc:4841] TS bec810c20d114ee795a11df6257d3d1a (127.21.83.2:34431): tablet 953f2928eba84e898c8d29e5aa8f1b76 (table test-table [id=ff08a46831cc4dc286318c8a0d99c48d]) successfully deleted
I20260807 07:02:01.145823 29639 ts_tablet_manager.cc:1905] T 953f2928eba84e898c8d29e5aa8f1b76 P cc9e4ec9d59b475cb2e1dd737d2855ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:02:01.245905 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 29027
I20260807 07:02:01.270159 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 29208
W20260807 07:02:01.270275 28926 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.83.1:42147 (error 108)
I20260807 07:02:01.273475 29598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:01.272973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:42147 (local address 127.0.0.1:57196)
0807 07:02:01.273142 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:01.273270 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.1:42147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260807 07:02:01.273876 29602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:01.271048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52006 (local address 127.21.83.2:34431)
0807 07:02:01.271475 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:01.271483 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:01.271506 (+ 23us) server_negotiation.cc:407] Connection header received
0807 07:02:01.271716 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:01.271723 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:01.271836 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:01.271943 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:01.273158 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:01.273165 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:01.273170 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:02:01.273191 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:01.273233 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:02:01.273250 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:01.273376 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:01.273516 (+ 140us) server_negotiation.cc:299] Negotiation successful
0807 07:02:01.273582 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269}
W20260807 07:02:01.273914 28926 catalog_manager.cc:4584] TS cc9e4ec9d59b475cb2e1dd737d2855ae (127.21.83.1:42147): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 953f2928eba84e898c8d29e5aa8f1b76: Network error: Client connection negotiation failed: client connection to 127.21.83.1:42147: connect: Connection refused (error 111)
I20260807 07:02:01.274610 29385 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:02:01.327605 29598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:01.327072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:42147 (local address 127.0.0.1:57198)
0807 07:02:01.327309 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:01.327419 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.1:42147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260807 07:02:01.329005 29598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:01.328550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:42147 (local address 127.0.0.1:57202)
0807 07:02:01.328723 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:01.328822 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.1:42147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260807 07:02:01.364800 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 29208
I20260807 07:02:01.389854 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 29394
I20260807 07:02:01.390296 29598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:01.389726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:42147 (local address 127.0.0.1:57210)
0807 07:02:01.390007 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:01.390114 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.1:42147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260807 07:02:01.391729 29598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:01.391320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:42147 (local address 127.0.0.1:57224)
0807 07:02:01.391482 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:01.391563 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.1:42147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260807 07:02:01.392838 29600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:01.390771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48070 (local address 127.21.83.3:45109)
0807 07:02:01.390961 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:01.390965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:01.390982 (+ 17us) server_negotiation.cc:407] Connection header received
0807 07:02:01.391146 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:01.391150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:01.391209 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:01.391298 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:01.392202 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:01.392207 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:01.392210 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:02:01.392226 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:01.392257 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:02:01.392269 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:01.392393 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:01.392485 (+ 92us) server_negotiation.cc:299] Negotiation successful
0807 07:02:01.392554 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260807 07:02:01.393638 29574 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:02:01.472394 29598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:01.471929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:42147 (local address 127.0.0.1:57238)
0807 07:02:01.472134 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:01.472230 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.1:42147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260807 07:02:01.473454 29598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:01.473066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:42147 (local address 127.0.0.1:57248)
0807 07:02:01.473259 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:01.473328 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.1:42147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260807 07:02:01.517745 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 29394
I20260807 07:02:01.541981 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 28904
I20260807 07:02:01.543141 28985 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:02:01.652284 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 28904
2026-08-07T07:02:01Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2642 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260807 07:02:01.696383 21836 mini_oidc.cc:152] Starting JWKS server
I20260807 07:02:01.699517 21836 webserver.cc:460] Webserver started at https://127.0.0.1:36801/ using document root <none> and password file <none>
I20260807 07:02:01.699908 21836 mini_oidc.cc:189] Starting OIDC Discovery server
I20260807 07:02:01.700630 21836 webserver.cc:460] Webserver started at http://127.0.0.1:35951/ using document root <none> and password file <none>
2026-08-07T07:02:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:02:01Z Disabled control of system clock
I20260807 07:02:01.737723 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:37041
--webserver_interface=127.21.83.62
--webserver_port=0
--builtin_ntp_servers=127.21.83.20:33437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:37041
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36801/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786086078080656-21836-0/testchainca.cert with env {}
W20260807 07:02:02.070013 29653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:02:02.070360 29653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:02:02.070500 29653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:02:02.078076 29653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:02:02.078239 29653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:02:02.078313 29653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:02:02.078397 29653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:02:02.078460 29653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260807 07:02:02.078552 29653 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260807 07:02:02.078608 29653 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36801/jwks.json
I20260807 07:02:02.089092 29653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:33437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:37041
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:37041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36801/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.83.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786086078080656-21836-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:02:02.090451 29653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:02:02.091769 29653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:02:02.103132 29665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:02.103205 29660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:02.103178 29663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:02:02.104017 29653 server_base.cc:1034] running on GCE node
I20260807 07:02:02.104720 29653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:02:02.106051 29653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:02:02.107261 29653 hybrid_clock.cc:648] HybridClock initialized: now 1786086122107219 us; error 44 us; skew 500 ppm
I20260807 07:02:02.107602 29653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 07:02:02.109732 29653 webserver.cc:460] Webserver started at http://127.21.83.62:35805/ using document root <none> and password file <none>
I20260807 07:02:02.110357 29653 fs_manager.cc:359] Metadata directory not provided
I20260807 07:02:02.110460 29653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:02:02.110750 29653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:02:02.113513 29653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "eb7fd6661f0c42fabb01875dd80cabef"
format_stamp: "Formatted at 2026-08-07 07:02:02 on dist-test-slave-ncp9"
server_key: "2724acdc399dde1826e3cf2205d5d999"
server_key_iv: "c855e9541f5a6e30afb0b50ceee68a27"
server_key_version: "encryptionkey@0"
I20260807 07:02:02.114253 29653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "eb7fd6661f0c42fabb01875dd80cabef"
format_stamp: "Formatted at 2026-08-07 07:02:02 on dist-test-slave-ncp9"
server_key: "2724acdc399dde1826e3cf2205d5d999"
server_key_iv: "c855e9541f5a6e30afb0b50ceee68a27"
server_key_version: "encryptionkey@0"
I20260807 07:02:02.119099 29653 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 07:02:02.122354 29671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:02.123574 29653 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 07:02:02.123732 29653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "eb7fd6661f0c42fabb01875dd80cabef"
format_stamp: "Formatted at 2026-08-07 07:02:02 on dist-test-slave-ncp9"
server_key: "2724acdc399dde1826e3cf2205d5d999"
server_key_iv: "c855e9541f5a6e30afb0b50ceee68a27"
server_key_version: "encryptionkey@0"
I20260807 07:02:02.123921 29653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:02:02.140202 29653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:02:02.141121 29653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:02:02.141413 29653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:02:02.159516 29653 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.62:37041
I20260807 07:02:02.159538 29750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.62:37041 every 8 connection(s)
I20260807 07:02:02.160981 29653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:02:02.165165 29751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:02:02.171200 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 29653
I20260807 07:02:02.171474 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:02:02.172253 21836 external_mini_cluster.cc:1428] Setting key 0d0e86f613b7f4320cc9e5082ffff3b3
I20260807 07:02:02.179436 29751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb7fd6661f0c42fabb01875dd80cabef: Bootstrap starting.
I20260807 07:02:02.183727 29751 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb7fd6661f0c42fabb01875dd80cabef: Neither blocks nor log segments found. Creating new log.
I20260807 07:02:02.185067 29751 log.cc:824] T 00000000000000000000000000000000 P eb7fd6661f0c42fabb01875dd80cabef: Log is configured to *not* fsync() on all Append() calls
I20260807 07:02:02.188781 29751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb7fd6661f0c42fabb01875dd80cabef: No bootstrap required, opened a new log
I20260807 07:02:02.189862 29759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:02.174749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49022 (local address 127.21.83.62:37041)
0807 07:02:02.176158 (+ 1409us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:02.176176 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:02.176309 (+ 133us) server_negotiation.cc:407] Connection header received
0807 07:02:02.178936 (+ 2627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:02.178970 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:02.179640 (+ 670us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:02.180144 (+ 504us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:02.180740 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:02.182829 (+ 2089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:02.184325 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:02.185424 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:02.185622 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:02.186880 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:02.186891 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:02.186910 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:02:02.187611 (+ 701us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:02.187727 (+ 116us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:02:02.187758 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:02.187882 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:02.188095 (+ 213us) server_negotiation.cc:299] Negotiation successful
0807 07:02:02.188453 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":776,"thread_start_us":472,"threads_started":1}
I20260807 07:02:02.193795 21836 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 07:02:02.197233 29751 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb7fd6661f0c42fabb01875dd80cabef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7fd6661f0c42fabb01875dd80cabef" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 37041 } }
I20260807 07:02:02.197585 29751 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb7fd6661f0c42fabb01875dd80cabef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:02:02.197685 29751 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb7fd6661f0c42fabb01875dd80cabef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb7fd6661f0c42fabb01875dd80cabef, State: Initialized, Role: FOLLOWER
I20260807 07:02:02.198278 29751 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb7fd6661f0c42fabb01875dd80cabef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7fd6661f0c42fabb01875dd80cabef" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 37041 } }
I20260807 07:02:02.198530 29751 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb7fd6661f0c42fabb01875dd80cabef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:02:02.198664 29751 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb7fd6661f0c42fabb01875dd80cabef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:02:02.198854 29751 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb7fd6661f0c42fabb01875dd80cabef [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:02:02.201206 29751 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb7fd6661f0c42fabb01875dd80cabef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7fd6661f0c42fabb01875dd80cabef" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 37041 } }
I20260807 07:02:02.201707 29751 leader_election.cc:302] T 00000000000000000000000000000000 P eb7fd6661f0c42fabb01875dd80cabef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb7fd6661f0c42fabb01875dd80cabef; no voters:
I20260807 07:02:02.202283 29751 leader_election.cc:288] T 00000000000000000000000000000000 P eb7fd6661f0c42fabb01875dd80cabef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 07:02:02.202440 29761 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb7fd6661f0c42fabb01875dd80cabef [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:02:02.203513 29761 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb7fd6661f0c42fabb01875dd80cabef [term 1 LEADER]: Becoming Leader. State: Replica: eb7fd6661f0c42fabb01875dd80cabef, State: Running, Role: LEADER
I20260807 07:02:02.204073 29761 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb7fd6661f0c42fabb01875dd80cabef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7fd6661f0c42fabb01875dd80cabef" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 37041 } }
I20260807 07:02:02.204453 29751 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb7fd6661f0c42fabb01875dd80cabef [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:02:02.206653 29762 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb7fd6661f0c42fabb01875dd80cabef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb7fd6661f0c42fabb01875dd80cabef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7fd6661f0c42fabb01875dd80cabef" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 37041 } } }
I20260807 07:02:02.207099 29762 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb7fd6661f0c42fabb01875dd80cabef [sys.catalog]: This master's current role is: LEADER
I20260807 07:02:02.208701 29774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:02:02.208650 29763 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb7fd6661f0c42fabb01875dd80cabef [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb7fd6661f0c42fabb01875dd80cabef. Latest consensus state: current_term: 1 leader_uuid: "eb7fd6661f0c42fabb01875dd80cabef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7fd6661f0c42fabb01875dd80cabef" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 37041 } } }
I20260807 07:02:02.209462 29763 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb7fd6661f0c42fabb01875dd80cabef [sys.catalog]: This master's current role is: LEADER
I20260807 07:02:02.213354 29774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:02:02.222307 29774 catalog_manager.cc:1347] Generated new cluster ID: ccadc9350cc849b783412f3d5119b3dc
I20260807 07:02:02.222429 29774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:02:02.238545 29774 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:02:02.239837 29774 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:02:02.250653 29774 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb7fd6661f0c42fabb01875dd80cabef: Generated new TSK 0
I20260807 07:02:02.251536 29774 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260807 07:02:02.286533 29796 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260807 07:02:02.286767 29796 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260807 07:02:02.289016 29759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:02.266787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49036 (local address 127.21.83.62:37041)
0807 07:02:02.266964 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:02.266969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:02.267635 (+ 666us) server_negotiation.cc:407] Connection header received
0807 07:02:02.268048 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:02.268053 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:02.268143 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:02.268276 (+ 133us) server_negotiation.cc:226] Negotiated authn=JWT
0807 07:02:02.269021 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:02.269807 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:02.271268 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:02.271732 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:02.271854 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:02.272449 (+ 595us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0807 07:02:02.288306 (+ 15857us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36801/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0807 07:02:02.288685 (+ 379us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49036: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36801/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":56}
W20260807 07:02:02.289155 29759 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49036: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36801/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260807 07:02:02.290542 29792 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:02.266753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:37041 (local address 127.0.0.1:49036)
0807 07:02:02.267444 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:02.267461 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:02.267759 (+ 298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:02.268339 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:02.268344 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:02.268358 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0807 07:02:02.268823 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:02.268838 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:02.269973 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:02.269978 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:02.271111 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:02.271124 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:02.271920 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:02.271927 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:02.272081 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:02.272151 (+ 70us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0807 07:02:02.288573 (+ 16422us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36801/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0807 07:02:02.288937 (+ 364us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.83.62:37041: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36801/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":355,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.21.83.62:37041: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36801/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260807 07:02:02.291929 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 29653
I20260807 07:02:02.293139 29745 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:02:02.386672 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 29653
2026-08-07T07:02:02Z chronyd exiting
I20260807 07:02:02.441340 21836 test_util.cc:175] -----------------------------------------------
I20260807 07:02:02.441478 21836 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786086078080656-21836-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (750 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260807 07:02:02.445338 21836 mini_oidc.cc:152] Starting JWKS server
I20260807 07:02:02.447093 21836 webserver.cc:460] Webserver started at https://127.0.0.1:35565/ using document root <none> and password file <none>
I20260807 07:02:02.447279 21836 mini_oidc.cc:189] Starting OIDC Discovery server
I20260807 07:02:02.447865 21836 webserver.cc:460] Webserver started at http://127.0.0.1:46493/ using document root <none> and password file <none>
2026-08-07T07:02:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:02:02Z Disabled control of system clock
I20260807 07:02:02.486212 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:45577
--webserver_interface=127.21.83.62
--webserver_port=0
--builtin_ntp_servers=127.21.83.20:33405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:45577
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35565/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786086078080656-21836-0/testchainca.cert with env {}
W20260807 07:02:02.825299 29806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:02:02.825572 29806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:02:02.825695 29806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:02:02.832829 29806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:02:02.832945 29806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:02:02.832996 29806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:02:02.833052 29806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:02:02.833096 29806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260807 07:02:02.833154 29806 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260807 07:02:02.833192 29806 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35565/jwks.json
I20260807 07:02:02.842921 29806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:33405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:45577
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:45577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35565/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.83.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786086078080656-21836-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:02:02.844046 29806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:02:02.845275 29806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:02:02.856866 29813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:02.857138 29812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:02.856913 29815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:02:02.857695 29806 server_base.cc:1034] running on GCE node
I20260807 07:02:02.858557 29806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:02:02.860149 29806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:02:02.861411 29806 hybrid_clock.cc:648] HybridClock initialized: now 1786086122861361 us; error 39 us; skew 500 ppm
I20260807 07:02:02.861927 29806 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 07:02:02.868940 29806 webserver.cc:460] Webserver started at http://127.21.83.62:45173/ using document root <none> and password file <none>
I20260807 07:02:02.869756 29806 fs_manager.cc:359] Metadata directory not provided
I20260807 07:02:02.869889 29806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:02:02.870237 29806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:02:02.873517 29806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "d0d9d6c130484b6ca3adc615c1c06097"
format_stamp: "Formatted at 2026-08-07 07:02:02 on dist-test-slave-ncp9"
server_key: "4c0d5a710f39c1ea92dd44df6d4678f5"
server_key_iv: "329a0b62f03b133951b0031e3caae545"
server_key_version: "encryptionkey@0"
I20260807 07:02:02.874383 29806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "d0d9d6c130484b6ca3adc615c1c06097"
format_stamp: "Formatted at 2026-08-07 07:02:02 on dist-test-slave-ncp9"
server_key: "4c0d5a710f39c1ea92dd44df6d4678f5"
server_key_iv: "329a0b62f03b133951b0031e3caae545"
server_key_version: "encryptionkey@0"
I20260807 07:02:02.880036 29806 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260807 07:02:02.883986 29822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:02.885761 29806 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 07:02:02.885980 29806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "d0d9d6c130484b6ca3adc615c1c06097"
format_stamp: "Formatted at 2026-08-07 07:02:02 on dist-test-slave-ncp9"
server_key: "4c0d5a710f39c1ea92dd44df6d4678f5"
server_key_iv: "329a0b62f03b133951b0031e3caae545"
server_key_version: "encryptionkey@0"
I20260807 07:02:02.886226 29806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:02:02.901491 29806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:02:02.902508 29806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:02:02.902835 29806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:02:02.922282 29806 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.62:45577
I20260807 07:02:02.922304 29896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.62:45577 every 8 connection(s)
I20260807 07:02:02.923769 29806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:02:02.927698 29897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:02:02.933111 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 29806
I20260807 07:02:02.933302 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:02:02.933941 21836 external_mini_cluster.cc:1428] Setting key 6627705b2513ebc0b8f76ef5476c52df
I20260807 07:02:02.938920 29897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0d9d6c130484b6ca3adc615c1c06097: Bootstrap starting.
I20260807 07:02:02.942443 29897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0d9d6c130484b6ca3adc615c1c06097: Neither blocks nor log segments found. Creating new log.
I20260807 07:02:02.943651 29897 log.cc:824] T 00000000000000000000000000000000 P d0d9d6c130484b6ca3adc615c1c06097: Log is configured to *not* fsync() on all Append() calls
I20260807 07:02:02.946970 29897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0d9d6c130484b6ca3adc615c1c06097: No bootstrap required, opened a new log
I20260807 07:02:02.948071 29902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:02.935852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58342 (local address 127.21.83.62:45577)
0807 07:02:02.936965 (+ 1113us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:02.936977 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:02.937030 (+ 53us) server_negotiation.cc:407] Connection header received
0807 07:02:02.938712 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:02.938735 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:02.939224 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:02.939560 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:02.940128 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:02.941501 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:02.943300 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:02.944039 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:02.944244 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:02.945330 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:02.945340 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:02.945356 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:02:02.946074 (+ 718us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:02.946174 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:02:02.946206 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:02.946332 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:02.946509 (+ 177us) server_negotiation.cc:299] Negotiation successful
0807 07:02:02.946834 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":401,"threads_started":1}
I20260807 07:02:02.952023 21836 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 07:02:02.957024 29897 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0d9d6c130484b6ca3adc615c1c06097 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d9d6c130484b6ca3adc615c1c06097" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45577 } }
I20260807 07:02:02.957370 29897 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0d9d6c130484b6ca3adc615c1c06097 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:02:02.957448 29897 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0d9d6c130484b6ca3adc615c1c06097 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0d9d6c130484b6ca3adc615c1c06097, State: Initialized, Role: FOLLOWER
I20260807 07:02:02.957922 29897 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0d9d6c130484b6ca3adc615c1c06097 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d9d6c130484b6ca3adc615c1c06097" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45577 } }
I20260807 07:02:02.958140 29897 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0d9d6c130484b6ca3adc615c1c06097 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:02:02.958240 29897 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0d9d6c130484b6ca3adc615c1c06097 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:02:02.958369 29897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0d9d6c130484b6ca3adc615c1c06097 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:02:02.960685 29897 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0d9d6c130484b6ca3adc615c1c06097 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d9d6c130484b6ca3adc615c1c06097" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45577 } }
I20260807 07:02:02.961180 29897 leader_election.cc:302] T 00000000000000000000000000000000 P d0d9d6c130484b6ca3adc615c1c06097 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0d9d6c130484b6ca3adc615c1c06097; no voters:
I20260807 07:02:02.961683 29897 leader_election.cc:288] T 00000000000000000000000000000000 P d0d9d6c130484b6ca3adc615c1c06097 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 07:02:02.961833 29907 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0d9d6c130484b6ca3adc615c1c06097 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:02:02.962559 29907 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0d9d6c130484b6ca3adc615c1c06097 [term 1 LEADER]: Becoming Leader. State: Replica: d0d9d6c130484b6ca3adc615c1c06097, State: Running, Role: LEADER
I20260807 07:02:02.963081 29907 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0d9d6c130484b6ca3adc615c1c06097 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d9d6c130484b6ca3adc615c1c06097" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45577 } }
I20260807 07:02:02.963650 29897 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0d9d6c130484b6ca3adc615c1c06097 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:02:02.965801 29908 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0d9d6c130484b6ca3adc615c1c06097 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0d9d6c130484b6ca3adc615c1c06097" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d9d6c130484b6ca3adc615c1c06097" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45577 } } }
I20260807 07:02:02.965979 29909 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0d9d6c130484b6ca3adc615c1c06097 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0d9d6c130484b6ca3adc615c1c06097. Latest consensus state: current_term: 1 leader_uuid: "d0d9d6c130484b6ca3adc615c1c06097" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d9d6c130484b6ca3adc615c1c06097" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45577 } } }
I20260807 07:02:02.966223 29908 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0d9d6c130484b6ca3adc615c1c06097 [sys.catalog]: This master's current role is: LEADER
I20260807 07:02:02.967001 29909 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0d9d6c130484b6ca3adc615c1c06097 [sys.catalog]: This master's current role is: LEADER
I20260807 07:02:02.967276 29912 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:02:02.972319 29912 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:02:02.981449 29912 catalog_manager.cc:1347] Generated new cluster ID: 4c1e3f431f764d2f809be2004d67cd29
I20260807 07:02:02.981571 29912 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:02:02.991665 29912 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:02:02.992980 29912 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:02:03.000306 29912 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0d9d6c130484b6ca3adc615c1c06097: Generated new TSK 0
I20260807 07:02:03.001214 29912 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260807 07:02:03.040485 29939 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260807 07:02:03.040611 29939 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260807 07:02:03.041586 29902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:03.015475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58356 (local address 127.21.83.62:45577)
0807 07:02:03.015642 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:03.015648 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:03.016081 (+ 433us) server_negotiation.cc:407] Connection header received
0807 07:02:03.016292 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:03.016296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:03.016365 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:03.016495 (+ 130us) server_negotiation.cc:226] Negotiated authn=JWT
0807 07:02:03.016983 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:03.017698 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:03.019069 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:03.019512 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:03.019625 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:03.020102 (+ 477us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0807 07:02:03.040947 (+ 20845us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35565/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0807 07:02:03.041287 (+ 340us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58356: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35565/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":48}
W20260807 07:02:03.041719 29902 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58356: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35565/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260807 07:02:03.041792 29938 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:03.015389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:45577 (local address 127.0.0.1:58356)
0807 07:02:03.015977 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:03.015996 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:03.016132 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:03.016537 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:03.016542 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:03.016556 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0807 07:02:03.016804 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:03.016814 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:03.017840 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:03.017844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:03.018924 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:03.018935 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:03.019705 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:03.019711 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:03.019843 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:03.019876 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0807 07:02:03.041150 (+ 21274us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35565/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0807 07:02:03.041383 (+ 233us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.83.62:45577: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35565/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":336,"threads_started":1}
I20260807 07:02:03.043493 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 29806
I20260807 07:02:03.044526 29892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:02:03.136276 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 29806
2026-08-07T07:02:03Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (750 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260807 07:02:03.195580 21836 mini_oidc.cc:152] Starting JWKS server
I20260807 07:02:03.197433 21836 webserver.cc:460] Webserver started at https://127.0.0.1:37183/ using document root <none> and password file <none>
I20260807 07:02:03.197669 21836 mini_oidc.cc:189] Starting OIDC Discovery server
I20260807 07:02:03.198300 21836 webserver.cc:460] Webserver started at http://127.0.0.1:35567/ using document root <none> and password file <none>
2026-08-07T07:02:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:02:03Z Disabled control of system clock
I20260807 07:02:03.233700 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:39393
--webserver_interface=127.21.83.62
--webserver_port=0
--builtin_ntp_servers=127.21.83.20:44491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:39393
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37183/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260807 07:02:03.555200 29946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:02:03.555538 29946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:02:03.555672 29946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:02:03.563616 29946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:02:03.563766 29946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:02:03.563834 29946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:02:03.563915 29946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:02:03.563992 29946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260807 07:02:03.564078 29946 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260807 07:02:03.564134 29946 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37183/jwks.json
I20260807 07:02:03.573871 29946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:44491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:39393
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:39393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37183/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:02:03.575233 29946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:02:03.577023 29946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:02:03.588148 29952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:03.588189 29953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:03.588543 29956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:02:03.589066 29946 server_base.cc:1034] running on GCE node
I20260807 07:02:03.589805 29946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:02:03.591085 29946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:02:03.592288 29946 hybrid_clock.cc:648] HybridClock initialized: now 1786086123592237 us; error 34 us; skew 500 ppm
I20260807 07:02:03.592607 29946 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 07:02:03.594673 29946 webserver.cc:460] Webserver started at http://127.21.83.62:44265/ using document root <none> and password file <none>
I20260807 07:02:03.595315 29946 fs_manager.cc:359] Metadata directory not provided
I20260807 07:02:03.595412 29946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:02:03.595705 29946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:02:03.598487 29946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "714c7cf9c1d04a7dad774ffa71c4827b"
format_stamp: "Formatted at 2026-08-07 07:02:03 on dist-test-slave-ncp9"
server_key: "7c37f19cf2d19a0b05c49791114762fa"
server_key_iv: "8138fb5be702612654d7d849de56ab41"
server_key_version: "encryptionkey@0"
I20260807 07:02:03.599246 29946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "714c7cf9c1d04a7dad774ffa71c4827b"
format_stamp: "Formatted at 2026-08-07 07:02:03 on dist-test-slave-ncp9"
server_key: "7c37f19cf2d19a0b05c49791114762fa"
server_key_iv: "8138fb5be702612654d7d849de56ab41"
server_key_version: "encryptionkey@0"
I20260807 07:02:03.603988 29946 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260807 07:02:03.607148 29963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:03.608608 29946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 07:02:03.608791 29946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "714c7cf9c1d04a7dad774ffa71c4827b"
format_stamp: "Formatted at 2026-08-07 07:02:03 on dist-test-slave-ncp9"
server_key: "7c37f19cf2d19a0b05c49791114762fa"
server_key_iv: "8138fb5be702612654d7d849de56ab41"
server_key_version: "encryptionkey@0"
I20260807 07:02:03.609068 29946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:02:03.622295 29946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:02:03.623085 29946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:02:03.623366 29946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:02:03.640563 29946 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.62:39393
I20260807 07:02:03.640585 30040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.62:39393 every 8 connection(s)
I20260807 07:02:03.641830 29946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:02:03.642387 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 29946
I20260807 07:02:03.642645 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:02:03.643411 21836 external_mini_cluster.cc:1428] Setting key 561ddbb6d8fbb0212feebdbb3b6d48d0
I20260807 07:02:03.646469 30041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:02:03.660205 30045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:03.645333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57046 (local address 127.21.83.62:39393)
0807 07:02:03.646477 (+ 1144us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:03.646493 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:03.646547 (+ 54us) server_negotiation.cc:407] Connection header received
0807 07:02:03.648840 (+ 2293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:03.648872 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:03.649551 (+ 679us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:03.650048 (+ 497us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:03.650639 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:03.652715 (+ 2076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:03.654172 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:03.655389 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:03.655610 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:03.656992 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:03.657009 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:03.657030 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:02:03.657834 (+ 804us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:03.657942 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:02:03.657986 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:03.658121 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:03.658392 (+ 271us) server_negotiation.cc:299] Negotiation successful
0807 07:02:03.658780 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":343,"threads_started":1}
I20260807 07:02:03.661438 30041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 714c7cf9c1d04a7dad774ffa71c4827b: Bootstrap starting.
I20260807 07:02:03.664674 21836 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 07:02:03.665277 30041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 714c7cf9c1d04a7dad774ffa71c4827b: Neither blocks nor log segments found. Creating new log.
I20260807 07:02:03.666330 30041 log.cc:824] T 00000000000000000000000000000000 P 714c7cf9c1d04a7dad774ffa71c4827b: Log is configured to *not* fsync() on all Append() calls
I20260807 07:02:03.669782 30041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 714c7cf9c1d04a7dad774ffa71c4827b: No bootstrap required, opened a new log
I20260807 07:02:03.677889 30041 raft_consensus.cc:348] T 00000000000000000000000000000000 P 714c7cf9c1d04a7dad774ffa71c4827b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "714c7cf9c1d04a7dad774ffa71c4827b" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39393 } }
I20260807 07:02:03.678301 30041 raft_consensus.cc:374] T 00000000000000000000000000000000 P 714c7cf9c1d04a7dad774ffa71c4827b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:02:03.678406 30041 raft_consensus.cc:729] T 00000000000000000000000000000000 P 714c7cf9c1d04a7dad774ffa71c4827b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 714c7cf9c1d04a7dad774ffa71c4827b, State: Initialized, Role: FOLLOWER
I20260807 07:02:03.679032 30041 consensus_queue.cc:260] T 00000000000000000000000000000000 P 714c7cf9c1d04a7dad774ffa71c4827b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "714c7cf9c1d04a7dad774ffa71c4827b" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39393 } }
I20260807 07:02:03.679270 30041 raft_consensus.cc:388] T 00000000000000000000000000000000 P 714c7cf9c1d04a7dad774ffa71c4827b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:02:03.679399 30041 raft_consensus.cc:482] T 00000000000000000000000000000000 P 714c7cf9c1d04a7dad774ffa71c4827b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:02:03.679579 30041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 714c7cf9c1d04a7dad774ffa71c4827b [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:02:03.682137 30041 raft_consensus.cc:504] T 00000000000000000000000000000000 P 714c7cf9c1d04a7dad774ffa71c4827b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "714c7cf9c1d04a7dad774ffa71c4827b" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39393 } }
I20260807 07:02:03.682829 30041 leader_election.cc:302] T 00000000000000000000000000000000 P 714c7cf9c1d04a7dad774ffa71c4827b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 714c7cf9c1d04a7dad774ffa71c4827b; no voters:
I20260807 07:02:03.683393 30041 leader_election.cc:288] T 00000000000000000000000000000000 P 714c7cf9c1d04a7dad774ffa71c4827b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 07:02:03.683542 30047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 714c7cf9c1d04a7dad774ffa71c4827b [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:02:03.684245 30047 raft_consensus.cc:686] T 00000000000000000000000000000000 P 714c7cf9c1d04a7dad774ffa71c4827b [term 1 LEADER]: Becoming Leader. State: Replica: 714c7cf9c1d04a7dad774ffa71c4827b, State: Running, Role: LEADER
I20260807 07:02:03.684814 30047 consensus_queue.cc:237] T 00000000000000000000000000000000 P 714c7cf9c1d04a7dad774ffa71c4827b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "714c7cf9c1d04a7dad774ffa71c4827b" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39393 } }
I20260807 07:02:03.685497 30041 sys_catalog.cc:564] T 00000000000000000000000000000000 P 714c7cf9c1d04a7dad774ffa71c4827b [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:02:03.687554 30048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 714c7cf9c1d04a7dad774ffa71c4827b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "714c7cf9c1d04a7dad774ffa71c4827b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "714c7cf9c1d04a7dad774ffa71c4827b" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39393 } } }
I20260807 07:02:03.687988 30048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 714c7cf9c1d04a7dad774ffa71c4827b [sys.catalog]: This master's current role is: LEADER
I20260807 07:02:03.688445 30049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 714c7cf9c1d04a7dad774ffa71c4827b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 714c7cf9c1d04a7dad774ffa71c4827b. Latest consensus state: current_term: 1 leader_uuid: "714c7cf9c1d04a7dad774ffa71c4827b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "714c7cf9c1d04a7dad774ffa71c4827b" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39393 } } }
I20260807 07:02:03.688843 30049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 714c7cf9c1d04a7dad774ffa71c4827b [sys.catalog]: This master's current role is: LEADER
I20260807 07:02:03.690057 30056 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:02:03.694712 30056 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:02:03.703323 30056 catalog_manager.cc:1347] Generated new cluster ID: 922edc22419b47bdb83aba2e447f15b0
I20260807 07:02:03.703455 30056 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:02:03.726089 30056 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:02:03.727978 30056 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:02:03.746917 30056 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 714c7cf9c1d04a7dad774ffa71c4827b: Generated new TSK 0
I20260807 07:02:03.747987 30056 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 07:02:03.782323 30045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:03.774158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57058 (local address 127.21.83.62:39393)
0807 07:02:03.774388 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:03.774394 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:03.774717 (+ 323us) server_negotiation.cc:407] Connection header received
0807 07:02:03.775022 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:03.775026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:03.775105 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:03.775257 (+ 152us) server_negotiation.cc:226] Negotiated authn=JWT
0807 07:02:03.775821 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:03.776534 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:03.778622 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:03.779108 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:03.779206 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:03.779788 (+ 582us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0807 07:02:03.781667 (+ 1879us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37183/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0807 07:02:03.781984 (+ 317us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57058: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37183/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":92}
W20260807 07:02:03.782541 30078 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:03.773903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39393 (local address 127.0.0.1:57058)
0807 07:02:03.774602 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:03.774623 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:03.774789 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:03.775294 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:03.775299 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:03.775313 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0807 07:02:03.775643 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:03.775655 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:03.776678 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:03.776683 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:03.778441 (+ 1758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:03.778458 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:03.779301 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:03.779308 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:03.779507 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:03.779556 (+ 49us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0807 07:02:03.781875 (+ 2319us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37183/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0807 07:02:03.782124 (+ 249us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.83.62:39393: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37183/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":389,"threads_started":1}
W20260807 07:02:03.782873 30045 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57058: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37183/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260807 07:02:03.783730 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 29946
I20260807 07:02:03.784833 30030 generic_service.cc:196] Checking for leaks (request via RPC)
E20260807 07:02:03.794276 30079 webserver.cc:533] Webserver: sslize failed:
E20260807 07:02:03.794369 30079 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260807 07:02:03.880350 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 29946
2026-08-07T07:02:03Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (744 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-07T07:02:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:02:03Z Disabled control of system clock
I20260807 07:02:03.974316 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:39695
--webserver_interface=127.21.83.62
--webserver_port=0
--builtin_ntp_servers=127.21.83.20:42757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:39695
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786086078080656-21836-0/testchainca.cert with env {}
W20260807 07:02:04.290547 30086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:02:04.290843 30086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:02:04.290979 30086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:02:04.297894 30086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:02:04.298007 30086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:02:04.298075 30086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:02:04.298157 30086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:02:04.298219 30086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260807 07:02:04.298301 30086 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260807 07:02:04.298354 30086 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260807 07:02:04.307925 30086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:42757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:39695
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:39695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.83.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786086078080656-21836-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:02:04.309365 30086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:02:04.310626 30086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:02:04.322083 30091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:04.322045 30093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:04.322012 30095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:02:04.322945 30086 server_base.cc:1034] running on GCE node
I20260807 07:02:04.323715 30086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:02:04.325150 30086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:02:04.326364 30086 hybrid_clock.cc:648] HybridClock initialized: now 1786086124326317 us; error 39 us; skew 500 ppm
I20260807 07:02:04.326707 30086 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 07:02:04.329035 30086 webserver.cc:460] Webserver started at http://127.21.83.62:40077/ using document root <none> and password file <none>
I20260807 07:02:04.329737 30086 fs_manager.cc:359] Metadata directory not provided
I20260807 07:02:04.329850 30086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:02:04.330165 30086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:02:04.333051 30086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "ce819f817e3247e8bf26ca8f3edffc80"
format_stamp: "Formatted at 2026-08-07 07:02:04 on dist-test-slave-ncp9"
server_key: "fcd59dc0f428f37d8211004087fd9ba0"
server_key_iv: "2b2ea935ed17e66791d7f85da3b94f95"
server_key_version: "encryptionkey@0"
I20260807 07:02:04.333846 30086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "ce819f817e3247e8bf26ca8f3edffc80"
format_stamp: "Formatted at 2026-08-07 07:02:04 on dist-test-slave-ncp9"
server_key: "fcd59dc0f428f37d8211004087fd9ba0"
server_key_iv: "2b2ea935ed17e66791d7f85da3b94f95"
server_key_version: "encryptionkey@0"
I20260807 07:02:04.338855 30086 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260807 07:02:04.342090 30101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:04.343327 30086 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 07:02:04.343492 30086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "ce819f817e3247e8bf26ca8f3edffc80"
format_stamp: "Formatted at 2026-08-07 07:02:04 on dist-test-slave-ncp9"
server_key: "fcd59dc0f428f37d8211004087fd9ba0"
server_key_iv: "2b2ea935ed17e66791d7f85da3b94f95"
server_key_version: "encryptionkey@0"
I20260807 07:02:04.343684 30086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:02:04.365805 30086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:02:04.366545 30086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:02:04.366793 30086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:02:04.383423 30086 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.62:39695
I20260807 07:02:04.383442 30175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.62:39695 every 8 connection(s)
I20260807 07:02:04.384585 30086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:02:04.387959 30181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:02:04.394907 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 30086
I20260807 07:02:04.395098 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:02:04.395900 21836 external_mini_cluster.cc:1428] Setting key d6ffb7eade02d957a83b2a6aadd7b18a
I20260807 07:02:04.398567 30181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce819f817e3247e8bf26ca8f3edffc80: Bootstrap starting.
I20260807 07:02:04.402498 30181 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce819f817e3247e8bf26ca8f3edffc80: Neither blocks nor log segments found. Creating new log.
I20260807 07:02:04.403649 30181 log.cc:824] T 00000000000000000000000000000000 P ce819f817e3247e8bf26ca8f3edffc80: Log is configured to *not* fsync() on all Append() calls
I20260807 07:02:04.407965 30181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce819f817e3247e8bf26ca8f3edffc80: No bootstrap required, opened a new log
I20260807 07:02:04.412940 30184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:04.398713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33372 (local address 127.21.83.62:39695)
0807 07:02:04.399878 (+ 1165us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:04.399891 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:04.399939 (+ 48us) server_negotiation.cc:407] Connection header received
0807 07:02:04.402304 (+ 2365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:04.402333 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:04.402953 (+ 620us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:04.404113 (+ 1160us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:04.405451 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:04.407288 (+ 1837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:04.408765 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:04.409527 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:04.409718 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:04.410771 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:04.410779 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:04.410792 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:02:04.411291 (+ 499us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:04.411361 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:02:04.411381 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:04.411482 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:04.411737 (+ 255us) server_negotiation.cc:299] Negotiation successful
0807 07:02:04.412023 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":409,"threads_started":1}
I20260807 07:02:04.415937 30181 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce819f817e3247e8bf26ca8f3edffc80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce819f817e3247e8bf26ca8f3edffc80" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39695 } }
I20260807 07:02:04.416263 30181 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce819f817e3247e8bf26ca8f3edffc80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:02:04.416356 30181 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce819f817e3247e8bf26ca8f3edffc80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce819f817e3247e8bf26ca8f3edffc80, State: Initialized, Role: FOLLOWER
I20260807 07:02:04.416486 21836 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 07:02:04.416867 30181 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce819f817e3247e8bf26ca8f3edffc80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce819f817e3247e8bf26ca8f3edffc80" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39695 } }
I20260807 07:02:04.417109 30181 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce819f817e3247e8bf26ca8f3edffc80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:02:04.417238 30181 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce819f817e3247e8bf26ca8f3edffc80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:02:04.417388 30181 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce819f817e3247e8bf26ca8f3edffc80 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:02:04.419402 30181 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce819f817e3247e8bf26ca8f3edffc80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce819f817e3247e8bf26ca8f3edffc80" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39695 } }
I20260807 07:02:04.419848 30181 leader_election.cc:302] T 00000000000000000000000000000000 P ce819f817e3247e8bf26ca8f3edffc80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce819f817e3247e8bf26ca8f3edffc80; no voters:
I20260807 07:02:04.420356 30181 leader_election.cc:288] T 00000000000000000000000000000000 P ce819f817e3247e8bf26ca8f3edffc80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 07:02:04.420467 30187 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce819f817e3247e8bf26ca8f3edffc80 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:02:04.421120 30187 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce819f817e3247e8bf26ca8f3edffc80 [term 1 LEADER]: Becoming Leader. State: Replica: ce819f817e3247e8bf26ca8f3edffc80, State: Running, Role: LEADER
I20260807 07:02:04.421602 30187 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce819f817e3247e8bf26ca8f3edffc80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce819f817e3247e8bf26ca8f3edffc80" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39695 } }
I20260807 07:02:04.422153 30181 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce819f817e3247e8bf26ca8f3edffc80 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:02:04.423985 30188 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce819f817e3247e8bf26ca8f3edffc80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce819f817e3247e8bf26ca8f3edffc80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce819f817e3247e8bf26ca8f3edffc80" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39695 } } }
I20260807 07:02:04.424302 30188 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce819f817e3247e8bf26ca8f3edffc80 [sys.catalog]: This master's current role is: LEADER
I20260807 07:02:04.425084 30194 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:02:04.426756 30189 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce819f817e3247e8bf26ca8f3edffc80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce819f817e3247e8bf26ca8f3edffc80. Latest consensus state: current_term: 1 leader_uuid: "ce819f817e3247e8bf26ca8f3edffc80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce819f817e3247e8bf26ca8f3edffc80" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39695 } } }
I20260807 07:02:04.427134 30189 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce819f817e3247e8bf26ca8f3edffc80 [sys.catalog]: This master's current role is: LEADER
I20260807 07:02:04.430120 30194 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:02:04.440210 30194 catalog_manager.cc:1347] Generated new cluster ID: 1e2d192955be49e780be586ccf475638
I20260807 07:02:04.440339 30194 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:02:04.483966 30194 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:02:04.485754 30194 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:02:04.502514 30194 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce819f817e3247e8bf26ca8f3edffc80: Generated new TSK 0
I20260807 07:02:04.503446 30194 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 07:02:04.989599 30184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:04.527982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33380 (local address 127.21.83.62:39695)
0807 07:02:04.528170 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:04.528179 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:04.528558 (+ 379us) server_negotiation.cc:407] Connection header received
0807 07:02:04.528789 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:04.528795 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:04.528883 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:04.529070 (+ 187us) server_negotiation.cc:226] Negotiated authn=JWT
0807 07:02:04.529454 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:04.530556 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:04.531923 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:04.532623 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:04.532745 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:04.533198 (+ 453us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0807 07:02:04.989023 (+455825us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0807 07:02:04.989355 (+ 332us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33380: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":57}
W20260807 07:02:04.989730 30184 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33380: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260807 07:02:04.989912 30226 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:04.527820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39695 (local address 127.0.0.1:33380)
0807 07:02:04.528466 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:04.528482 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:04.528618 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:04.529070 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:04.529074 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:04.529084 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0807 07:02:04.529317 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:04.529325 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:04.530711 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:04.530715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:04.531789 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:04.531800 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:04.532766 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:04.532770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:04.532886 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:04.532957 (+ 71us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0807 07:02:04.989319 (+456362us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0807 07:02:04.989576 (+ 257us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.83.62:39695: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":355,"threads_started":1}
I20260807 07:02:04.991185 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 30086
I20260807 07:02:04.992331 30168 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:02:05.097231 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 30086
2026-08-07T07:02:05Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1199 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (418 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (429 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 07:02:06 dist-test-slave-ncp9 krb5kdc[30246](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 07:02:06 dist-test-slave-ncp9 krb5kdc[30246](info): set up 2 sockets
Aug 07 07:02:06 dist-test-slave-ncp9 krb5kdc[30246](info): commencing operation
krb5kdc: starting...
W20260807 07:02:07.147490 21836 mini_kdc.cc:121] Time spent starting KDC: real 1.158s user 0.000s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Aug 07 07:02:07 dist-test-slave-ncp9 krb5kdc[30246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T07:02:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:02:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
I20260807 07:02:07.379760 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:38371
--webserver_interface=127.21.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:36155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:38371
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:02:07.709329 30263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:02:07.709616 30263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:02:07.709728 30263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:02:07.716866 30263 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:02:07.717020 30263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:02:07.717113 30263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:02:07.717193 30263 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:02:07.717257 30263 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:02:07.727337 30263 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:36155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:38371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:38371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:02:07.728696 30263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:02:07.730022 30263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:02:07.741609 30269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:07.741640 30272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:07.741842 30270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:02:07.742923 30263 server_base.cc:1034] running on GCE node
I20260807 07:02:07.743673 30263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:02:07.745074 30263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:02:07.746265 30263 hybrid_clock.cc:648] HybridClock initialized: now 1786086127746216 us; error 42 us; skew 500 ppm
I20260807 07:02:07.746595 30263 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:02:07 dist-test-slave-ncp9 krb5kdc[30246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086127, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:02:07.749724 30263 init.cc:377] Logged in from keytab as kudu/127.21.83.62@KRBTEST.COM (short username kudu)
I20260807 07:02:07.751397 30263 webserver.cc:460] Webserver started at http://127.21.83.62:38975/ using document root <none> and password file <none>
I20260807 07:02:07.752142 30263 fs_manager.cc:359] Metadata directory not provided
I20260807 07:02:07.752274 30263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:02:07.752599 30263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:02:07.755780 30263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "dbc1e5f08032455da36884f3d1d4f275"
format_stamp: "Formatted at 2026-08-07 07:02:07 on dist-test-slave-ncp9"
server_key: "817a14c6d8391eb8ed0bfbe7a1a234c5"
server_key_iv: "e05de6cf62dfff3c5071b42469d4856e"
server_key_version: "encryptionkey@0"
I20260807 07:02:07.756608 30263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "dbc1e5f08032455da36884f3d1d4f275"
format_stamp: "Formatted at 2026-08-07 07:02:07 on dist-test-slave-ncp9"
server_key: "817a14c6d8391eb8ed0bfbe7a1a234c5"
server_key_iv: "e05de6cf62dfff3c5071b42469d4856e"
server_key_version: "encryptionkey@0"
I20260807 07:02:07.762066 30263 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260807 07:02:07.765578 30280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:07.766803 30263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 07:02:07.766960 30263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "dbc1e5f08032455da36884f3d1d4f275"
format_stamp: "Formatted at 2026-08-07 07:02:07 on dist-test-slave-ncp9"
server_key: "817a14c6d8391eb8ed0bfbe7a1a234c5"
server_key_iv: "e05de6cf62dfff3c5071b42469d4856e"
server_key_version: "encryptionkey@0"
I20260807 07:02:07.767154 30263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:02:07.788133 30263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:02:07.791734 30263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:02:07.791987 30263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:02:07.809654 30263 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.62:38371
I20260807 07:02:07.809702 30356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.62:38371 every 8 connection(s)
I20260807 07:02:07.810976 30263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:02:07.812584 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 30263
I20260807 07:02:07.812763 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:02:07.813421 21836 external_mini_cluster.cc:1428] Setting key ab503eecf2133492c721d1cd8b881eef
I20260807 07:02:07.814956 30357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 07:02:07 dist-test-slave-ncp9 krb5kdc[30246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:02:07.828507 30357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbc1e5f08032455da36884f3d1d4f275: Bootstrap starting.
I20260807 07:02:07.831553 30357 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbc1e5f08032455da36884f3d1d4f275: Neither blocks nor log segments found. Creating new log.
I20260807 07:02:07.832752 30357 log.cc:824] T 00000000000000000000000000000000 P dbc1e5f08032455da36884f3d1d4f275: Log is configured to *not* fsync() on all Append() calls
I20260807 07:02:07.835424 30357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbc1e5f08032455da36884f3d1d4f275: No bootstrap required, opened a new log
I20260807 07:02:07.835738 30361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:07.815317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46986 (local address 127.21.83.62:38371)
0807 07:02:07.816109 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:07.816120 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:07.816159 (+ 39us) server_negotiation.cc:407] Connection header received
0807 07:02:07.817812 (+ 1653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:07.817834 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:07.818205 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:07.818523 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:07.820302 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:07.821667 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:07.823371 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:07.824006 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:07.824134 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:07.828875 (+ 4741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:07.828883 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:07.828922 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:07.828971 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:07.831860 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:07.832643 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:07.832650 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:07.832655 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:07.832830 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:07.833236 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:07.833240 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:07.833242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:07.833740 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:07.834041 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:07.834455 (+ 414us) server_negotiation.cc:299] Negotiation successful
0807 07:02:07.834830 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":339,"threads_started":1}
I20260807 07:02:07.842602 30357 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbc1e5f08032455da36884f3d1d4f275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbc1e5f08032455da36884f3d1d4f275" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 38371 } }
I20260807 07:02:07.842968 30357 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbc1e5f08032455da36884f3d1d4f275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:02:07.843062 30357 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbc1e5f08032455da36884f3d1d4f275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbc1e5f08032455da36884f3d1d4f275, State: Initialized, Role: FOLLOWER
I20260807 07:02:07.843580 30357 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbc1e5f08032455da36884f3d1d4f275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbc1e5f08032455da36884f3d1d4f275" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 38371 } }
I20260807 07:02:07.843860 30357 raft_consensus.cc:388] T 00000000000000000000000000000000 P dbc1e5f08032455da36884f3d1d4f275 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:02:07.844022 30357 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbc1e5f08032455da36884f3d1d4f275 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:02:07.844215 30357 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbc1e5f08032455da36884f3d1d4f275 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:02:07.846375 30357 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbc1e5f08032455da36884f3d1d4f275 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbc1e5f08032455da36884f3d1d4f275" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 38371 } }
I20260807 07:02:07.847023 30357 leader_election.cc:302] T 00000000000000000000000000000000 P dbc1e5f08032455da36884f3d1d4f275 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbc1e5f08032455da36884f3d1d4f275; no voters:
I20260807 07:02:07.847681 30357 leader_election.cc:288] T 00000000000000000000000000000000 P dbc1e5f08032455da36884f3d1d4f275 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 07:02:07.847776 30364 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbc1e5f08032455da36884f3d1d4f275 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:02:07.848421 30364 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbc1e5f08032455da36884f3d1d4f275 [term 1 LEADER]: Becoming Leader. State: Replica: dbc1e5f08032455da36884f3d1d4f275, State: Running, Role: LEADER
I20260807 07:02:07.849012 30364 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbc1e5f08032455da36884f3d1d4f275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbc1e5f08032455da36884f3d1d4f275" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 38371 } }
I20260807 07:02:07.850003 30357 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbc1e5f08032455da36884f3d1d4f275 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:02:07.852025 30365 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbc1e5f08032455da36884f3d1d4f275 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbc1e5f08032455da36884f3d1d4f275" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbc1e5f08032455da36884f3d1d4f275" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 38371 } } }
I20260807 07:02:07.852434 30365 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbc1e5f08032455da36884f3d1d4f275 [sys.catalog]: This master's current role is: LEADER
I20260807 07:02:07.853226 30366 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbc1e5f08032455da36884f3d1d4f275 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbc1e5f08032455da36884f3d1d4f275. Latest consensus state: current_term: 1 leader_uuid: "dbc1e5f08032455da36884f3d1d4f275" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbc1e5f08032455da36884f3d1d4f275" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 38371 } } }
I20260807 07:02:07.853668 30366 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbc1e5f08032455da36884f3d1d4f275 [sys.catalog]: This master's current role is: LEADER
I20260807 07:02:07.854133 30371 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:02:07.861037 30371 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:02:07.873236 30371 catalog_manager.cc:1347] Generated new cluster ID: b999f855172244f6b7237a5dc697f55d
I20260807 07:02:07.873389 30371 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:02:07.890895 30371 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:02:07.892302 30371 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:02:07.903707 30371 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbc1e5f08032455da36884f3d1d4f275: Generated new TSK 0
I20260807 07:02:07.904476 30371 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.83.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab.
Entry for principal kudu/127.21.83.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab.
I20260807 07:02:07.969862 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.1:0
--local_ip_for_outbound_sockets=127.21.83.1
--webserver_interface=127.21.83.1
--webserver_port=0
--tserver_master_addrs=127.21.83.62:38371
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab
--principal=kudu/127.21.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:36155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:02:08.303531 30390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:02:08.303800 30390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:02:08.303894 30390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:02:08.310941 30390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:02:08.311074 30390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:02:08.311203 30390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.1
I20260807 07:02:08.320832 30390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:36155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab
--principal=kudu/127.21.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.83.1
--webserver_port=0
--tserver_master_addrs=127.21.83.62:38371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.1
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:02:08.322129 30390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:02:08.323405 30390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:02:08.336072 30397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:08.336114 30400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:08.336510 30402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:02:08.337170 30390 server_base.cc:1034] running on GCE node
I20260807 07:02:08.337802 30390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:02:08.338773 30390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:02:08.339975 30390 hybrid_clock.cc:648] HybridClock initialized: now 1786086128339923 us; error 47 us; skew 500 ppm
I20260807 07:02:08.340288 30390 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:02:08 dist-test-slave-ncp9 krb5kdc[30246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086128, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:02:08.347990 30390 init.cc:377] Logged in from keytab as kudu/127.21.83.1@KRBTEST.COM (short username kudu)
I20260807 07:02:08.349558 30390 webserver.cc:460] Webserver started at http://127.21.83.1:44801/ using document root <none> and password file <none>
I20260807 07:02:08.350258 30390 fs_manager.cc:359] Metadata directory not provided
I20260807 07:02:08.350381 30390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:02:08.350714 30390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:02:08.353765 30390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/ts-0/data/instance:
uuid: "311e79550fc247f4940fc57994cd3af9"
format_stamp: "Formatted at 2026-08-07 07:02:08 on dist-test-slave-ncp9"
server_key: "69c0264cd4e69321c4a4d7b9e4fd323c"
server_key_iv: "73fb88128b989f452556016e2a8b9e0d"
server_key_version: "encryptionkey@0"
I20260807 07:02:08.354549 30390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/ts-0/wal/instance:
uuid: "311e79550fc247f4940fc57994cd3af9"
format_stamp: "Formatted at 2026-08-07 07:02:08 on dist-test-slave-ncp9"
server_key: "69c0264cd4e69321c4a4d7b9e4fd323c"
server_key_iv: "73fb88128b989f452556016e2a8b9e0d"
server_key_version: "encryptionkey@0"
I20260807 07:02:08.360201 30390 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 07:02:08.366238 30411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:08.367935 30390 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260807 07:02:08.368153 30390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/ts-0/wal
uuid: "311e79550fc247f4940fc57994cd3af9"
format_stamp: "Formatted at 2026-08-07 07:02:08 on dist-test-slave-ncp9"
server_key: "69c0264cd4e69321c4a4d7b9e4fd323c"
server_key_iv: "73fb88128b989f452556016e2a8b9e0d"
server_key_version: "encryptionkey@0"
I20260807 07:02:08.368387 30390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:02:08.387912 30390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:02:08.391923 30390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:02:08.392189 30390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:02:08.393181 30390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:02:08.394507 30390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:02:08.394596 30390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:08.394719 30390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:02:08.394774 30390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:08.425019 30390 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.1:46431
I20260807 07:02:08.425063 30563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.1:46431 every 8 connection(s)
I20260807 07:02:08.426311 30390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
I20260807 07:02:08.433374 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 30390
I20260807 07:02:08.433562 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/ts-0/wal/instance
I20260807 07:02:08.434206 21836 external_mini_cluster.cc:1428] Setting key 43ea0c66feccb90bee8efd93ced71816
Aug 07 07:02:08 dist-test-slave-ncp9 krb5kdc[30246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086128, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.1@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:02:08.446105 30566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:08.428879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:57497 (local address 127.21.83.62:38371)
0807 07:02:08.429391 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:08.429395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:08.430189 (+ 794us) server_negotiation.cc:407] Connection header received
0807 07:02:08.432001 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:08.432007 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:08.432087 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:08.432195 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:08.433886 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:08.434646 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:08.436458 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:08.436858 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:08.436997 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:08.440405 (+ 3408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:08.440411 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:08.440414 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:08.440448 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:08.442977 (+ 2529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:08.443701 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:08.443707 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:08.443710 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:08.443855 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:08.444312 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:08.444317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:08.444319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:08.444662 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:08.444805 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:08.445517 (+ 712us) server_negotiation.cc:299] Negotiation successful
0807 07:02:08.445737 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":266,"threads_started":1}
I20260807 07:02:08.447077 30568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:08.429268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38371 (local address 127.21.83.1:57497)
0807 07:02:08.430011 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:08.430051 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:08.431726 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:08.432455 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:08.432464 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:08.433071 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:08.433707 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:08.433728 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:08.434811 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:08.434815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:08.436310 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:08.436324 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:08.437101 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:08.437109 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:08.437331 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:08.437990 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:08.438018 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:08.440185 (+ 2167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:08.443113 (+ 2928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:08.443123 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:08.443141 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:08.443575 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:08.443988 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:08.443992 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:08.443994 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:08.444195 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:08.444779 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:08.444785 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:08.445161 (+ 376us) client_negotiation.cc:769] Sending connection context
0807 07:02:08.445396 (+ 235us) client_negotiation.cc:240] Negotiation successful
0807 07:02:08.445799 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":309,"threads_started":1}
I20260807 07:02:08.448689 30564 heartbeater.cc:347] Connected to a master server at 127.21.83.62:38371
I20260807 07:02:08.449007 30564 heartbeater.cc:464] Registering TS with master...
I20260807 07:02:08.449714 30564 heartbeater.cc:511] Master 127.21.83.62:38371 requested a full tablet report, sending...
I20260807 07:02:08.452020 30302 ts_manager.cc:194] Registered new tserver with Master: 311e79550fc247f4940fc57994cd3af9 (127.21.83.1:46431)
I20260807 07:02:08.453905 30302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.1@KRBTEST.COM'} at 127.21.83.1:57497
WARNING: no policy specified for kudu/127.21.83.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab.
Entry for principal kudu/127.21.83.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab.
I20260807 07:02:08.487653 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.2:0
--local_ip_for_outbound_sockets=127.21.83.2
--webserver_interface=127.21.83.2
--webserver_port=0
--tserver_master_addrs=127.21.83.62:38371
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab
--principal=kudu/127.21.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:36155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:02:08.820205 30577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:02:08.820466 30577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:02:08.820614 30577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:02:08.827613 30577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:02:08.827718 30577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:02:08.827858 30577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.2
I20260807 07:02:08.837044 30577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:36155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab
--principal=kudu/127.21.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.83.2
--webserver_port=0
--tserver_master_addrs=127.21.83.62:38371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.2
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:02:08.838104 30577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:02:08.839365 30577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:02:08.853336 30590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:08.853374 30586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:08.853294 30588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:02:08.854250 30577 server_base.cc:1034] running on GCE node
I20260807 07:02:08.854920 30577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:02:08.856050 30577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:02:08.857254 30577 hybrid_clock.cc:648] HybridClock initialized: now 1786086128857206 us; error 45 us; skew 500 ppm
I20260807 07:02:08.857582 30577 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:02:08 dist-test-slave-ncp9 krb5kdc[30246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086128, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:02:08.860845 30577 init.cc:377] Logged in from keytab as kudu/127.21.83.2@KRBTEST.COM (short username kudu)
I20260807 07:02:08.862835 30577 webserver.cc:460] Webserver started at http://127.21.83.2:34503/ using document root <none> and password file <none>
I20260807 07:02:08.863658 30577 fs_manager.cc:359] Metadata directory not provided
I20260807 07:02:08.863811 30577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:02:08.864142 30577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:02:08.867551 30577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/ts-1/data/instance:
uuid: "77f9df0bacf2422c8775acc2a0602a9e"
format_stamp: "Formatted at 2026-08-07 07:02:08 on dist-test-slave-ncp9"
server_key: "5e0b1e76de97d2b0a8ddfbfc42458435"
server_key_iv: "0d3016e7755d51112543d1db6dba9def"
server_key_version: "encryptionkey@0"
I20260807 07:02:08.868441 30577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/ts-1/wal/instance:
uuid: "77f9df0bacf2422c8775acc2a0602a9e"
format_stamp: "Formatted at 2026-08-07 07:02:08 on dist-test-slave-ncp9"
server_key: "5e0b1e76de97d2b0a8ddfbfc42458435"
server_key_iv: "0d3016e7755d51112543d1db6dba9def"
server_key_version: "encryptionkey@0"
I20260807 07:02:08.874306 30577 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.001s
I20260807 07:02:08.878085 30600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:08.879475 30577 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 07:02:08.879638 30577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/ts-1/wal
uuid: "77f9df0bacf2422c8775acc2a0602a9e"
format_stamp: "Formatted at 2026-08-07 07:02:08 on dist-test-slave-ncp9"
server_key: "5e0b1e76de97d2b0a8ddfbfc42458435"
server_key_iv: "0d3016e7755d51112543d1db6dba9def"
server_key_version: "encryptionkey@0"
I20260807 07:02:08.879840 30577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:02:08.896574 30577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:02:08.900852 30577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:02:08.901163 30577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:02:08.902163 30577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:02:08.903550 30577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:02:08.903632 30577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:08.903784 30577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:02:08.903839 30577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:08.936113 30577 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.2:46059
I20260807 07:02:08.936163 30763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.2:46059 every 8 connection(s)
I20260807 07:02:08.937438 30577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
I20260807 07:02:08.947808 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 30577
I20260807 07:02:08.948009 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/ts-1/wal/instance
I20260807 07:02:08.948462 21836 external_mini_cluster.cc:1428] Setting key 7421345cf4bdf89a82f7d1d6686fae1f
Aug 07 07:02:08 dist-test-slave-ncp9 krb5kdc[30246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086128, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.2@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:02:08.958169 30566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:08.939855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:55299 (local address 127.21.83.62:38371)
0807 07:02:08.940030 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:08.940034 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:08.941227 (+ 1193us) server_negotiation.cc:407] Connection header received
0807 07:02:08.943033 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:08.943039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:08.943117 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:08.943213 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:08.944687 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:08.945488 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:08.947223 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:08.947688 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:08.947808 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:08.951874 (+ 4066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:08.951880 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:08.951884 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:08.951923 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:08.954638 (+ 2715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:08.955518 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:08.955523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:08.955526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:08.955693 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:08.956174 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:08.956188 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:08.956190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:08.956485 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:08.956629 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:08.957526 (+ 897us) server_negotiation.cc:299] Negotiation successful
0807 07:02:08.957812 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260807 07:02:08.959293 30766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:08.940296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38371 (local address 127.21.83.2:55299)
0807 07:02:08.941071 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:08.941117 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:08.942771 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:08.943448 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:08.943456 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:08.943972 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:08.944518 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:08.944536 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:08.945655 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:08.945659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:08.947046 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:08.947061 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:08.947929 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:08.947937 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:08.948206 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:08.948935 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:08.948967 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:08.951606 (+ 2639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:08.954830 (+ 3224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:08.954842 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:08.954858 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:08.955379 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:08.955815 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:08.955821 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:08.955825 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:08.956056 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:08.956623 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:08.956631 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:08.957133 (+ 502us) client_negotiation.cc:769] Sending connection context
0807 07:02:08.957415 (+ 282us) client_negotiation.cc:240] Negotiation successful
0807 07:02:08.958043 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":300,"threads_started":1}
I20260807 07:02:08.960989 30764 heartbeater.cc:347] Connected to a master server at 127.21.83.62:38371
I20260807 07:02:08.961337 30764 heartbeater.cc:464] Registering TS with master...
I20260807 07:02:08.963689 30764 heartbeater.cc:511] Master 127.21.83.62:38371 requested a full tablet report, sending...
I20260807 07:02:08.965938 30302 ts_manager.cc:194] Registered new tserver with Master: 77f9df0bacf2422c8775acc2a0602a9e (127.21.83.2:46059)
I20260807 07:02:08.967145 30302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.2@KRBTEST.COM'} at 127.21.83.2:55299
WARNING: no policy specified for kudu/127.21.83.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab.
Entry for principal kudu/127.21.83.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab.
I20260807 07:02:09.008760 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.3:0
--local_ip_for_outbound_sockets=127.21.83.3
--webserver_interface=127.21.83.3
--webserver_port=0
--tserver_master_addrs=127.21.83.62:38371
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab
--principal=kudu/127.21.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:36155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:02:09.341068 30772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:02:09.341331 30772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:02:09.341423 30772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:02:09.348639 30772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:02:09.348742 30772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:02:09.348855 30772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.3
I20260807 07:02:09.358553 30772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:36155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab
--principal=kudu/127.21.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.83.3
--webserver_port=0
--tserver_master_addrs=127.21.83.62:38371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.3
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:02:09.359885 30772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:02:09.361294 30772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:02:09.377209 30779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:09.377256 30780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:09.377441 30782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:02:09.378293 30772 server_base.cc:1034] running on GCE node
I20260807 07:02:09.378954 30772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:02:09.379985 30772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:02:09.381189 30772 hybrid_clock.cc:648] HybridClock initialized: now 1786086129381137 us; error 47 us; skew 500 ppm
I20260807 07:02:09.381520 30772 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:02:09 dist-test-slave-ncp9 krb5kdc[30246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086129, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:02:09.385260 30772 init.cc:377] Logged in from keytab as kudu/127.21.83.3@KRBTEST.COM (short username kudu)
I20260807 07:02:09.386941 30772 webserver.cc:460] Webserver started at http://127.21.83.3:43349/ using document root <none> and password file <none>
I20260807 07:02:09.387696 30772 fs_manager.cc:359] Metadata directory not provided
I20260807 07:02:09.387821 30772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:02:09.388144 30772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:02:09.391299 30772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/ts-2/data/instance:
uuid: "386d17248eae42918abb2a3be0647086"
format_stamp: "Formatted at 2026-08-07 07:02:09 on dist-test-slave-ncp9"
server_key: "fa3fcb7d5c74a9480344145aa92d8825"
server_key_iv: "f7d57c6308a1540061c989c5782c1bdb"
server_key_version: "encryptionkey@0"
I20260807 07:02:09.392100 30772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/ts-2/wal/instance:
uuid: "386d17248eae42918abb2a3be0647086"
format_stamp: "Formatted at 2026-08-07 07:02:09 on dist-test-slave-ncp9"
server_key: "fa3fcb7d5c74a9480344145aa92d8825"
server_key_iv: "f7d57c6308a1540061c989c5782c1bdb"
server_key_version: "encryptionkey@0"
I20260807 07:02:09.397317 30772 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 07:02:09.400547 30791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:09.401773 30772 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 07:02:09.401911 30772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/ts-2/wal
uuid: "386d17248eae42918abb2a3be0647086"
format_stamp: "Formatted at 2026-08-07 07:02:09 on dist-test-slave-ncp9"
server_key: "fa3fcb7d5c74a9480344145aa92d8825"
server_key_iv: "f7d57c6308a1540061c989c5782c1bdb"
server_key_version: "encryptionkey@0"
I20260807 07:02:09.402098 30772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:02:09.416978 30772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:02:09.422488 30772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:02:09.422765 30772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:02:09.423748 30772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:02:09.425262 30772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:02:09.425338 30772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:09.425443 30772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:02:09.425494 30772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:09.457029 30564 heartbeater.cc:503] Master 127.21.83.62:38371 was elected leader, sending a full tablet report...
I20260807 07:02:09.464613 30772 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.3:36141
I20260807 07:02:09.464663 30946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.3:36141 every 8 connection(s)
I20260807 07:02:09.465879 30772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
I20260807 07:02:09.475822 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 30772
I20260807 07:02:09.476004 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/minicluster-data/ts-2/wal/instance
I20260807 07:02:09.476536 21836 external_mini_cluster.cc:1428] Setting key d015e157765e8362296e3e708307a20f
Aug 07 07:02:09 dist-test-slave-ncp9 krb5kdc[30246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086129, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.3@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:02:09.484330 30950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:09.468194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:57819 (local address 127.21.83.62:38371)
0807 07:02:09.468681 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:09.468687 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:09.469577 (+ 890us) server_negotiation.cc:407] Connection header received
0807 07:02:09.471372 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:09.471377 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:09.471449 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:09.471541 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:09.472995 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:09.473818 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:09.475235 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:09.475614 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:09.475698 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:09.479339 (+ 3641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:09.479343 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:09.479347 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:09.479377 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:09.481752 (+ 2375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:09.482402 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:09.482405 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:09.482407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:09.482529 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:09.482878 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:09.482881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:09.482883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:09.483155 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:09.483261 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:09.483904 (+ 643us) server_negotiation.cc:299] Negotiation successful
0807 07:02:09.484065 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":230,"threads_started":1}
I20260807 07:02:09.485261 30951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:09.468684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38371 (local address 127.21.83.3:57819)
0807 07:02:09.469430 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:09.469470 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:09.471126 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:09.471735 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:09.471743 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:09.472280 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:09.472831 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:09.472849 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:09.473943 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:09.473954 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:09.475125 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:09.475137 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:09.475826 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:09.475833 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:09.476120 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:09.476805 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:09.476835 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:09.479154 (+ 2319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:09.481886 (+ 2732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:09.481892 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:09.481902 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:09.482312 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:09.482603 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:09.482606 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:09.482608 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:09.482779 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:09.483251 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:09.483256 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:09.483568 (+ 312us) client_negotiation.cc:769] Sending connection context
0807 07:02:09.483799 (+ 231us) client_negotiation.cc:240] Negotiation successful
0807 07:02:09.484138 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":317,"threads_started":1}
I20260807 07:02:09.486455 30947 heartbeater.cc:347] Connected to a master server at 127.21.83.62:38371
I20260807 07:02:09.486712 30947 heartbeater.cc:464] Registering TS with master...
I20260807 07:02:09.487363 30947 heartbeater.cc:511] Master 127.21.83.62:38371 requested a full tablet report, sending...
I20260807 07:02:09.488732 30302 ts_manager.cc:194] Registered new tserver with Master: 386d17248eae42918abb2a3be0647086 (127.21.83.3:36141)
I20260807 07:02:09.489517 30302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.3@KRBTEST.COM'} at 127.21.83.3:57819
I20260807 07:02:09.491660 21836 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/krb5kdc/test-admin.keytab.
Aug 07 07:02:09 dist-test-slave-ncp9 krb5kdc[30246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:02:09.521940 21836 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260807 07:02:09.526582 30960 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:09.524375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36692 (local address 127.21.83.1:46431)
0807 07:02:09.524997 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:09.525005 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:09.525033 (+ 28us) server_negotiation.cc:407] Connection header received
0807 07:02:09.525094 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:09.525101 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:09.525317 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:09.525439 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:09.526217 (+ 778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36692: BlockingRecv error: recv got EOF from 127.0.0.1:36692 (error 108)
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":289,"threads_started":1}
W20260807 07:02:09.526656 21836 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/krb5kdc/krb5cc' of type FILE
Aug 07 07:02:09 dist-test-slave-ncp9 krb5kdc[30246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:02:09.528313 21836 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 07 07:02:09 dist-test-slave-ncp9 krb5kdc[30246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:02:09.529569 21836 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 07 07:02:09 dist-test-slave-ncp9 krb5kdc[30246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.1@KRBTEST.COM
I20260807 07:02:09.543736 30960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:09.530974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36706 (local address 127.21.83.1:46431)
0807 07:02:09.531161 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:09.531166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:09.531364 (+ 198us) server_negotiation.cc:407] Connection header received
0807 07:02:09.531505 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:09.531509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:09.531563 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:09.531636 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:09.532536 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:09.533620 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:09.534790 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:09.535337 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:09.535440 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:09.538680 (+ 3240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:09.538691 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:09.538703 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:09.538734 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:09.541514 (+ 2780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:09.542044 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:09.542050 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:09.542052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:09.542163 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:09.542438 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:09.542441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:09.542443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:09.542776 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:09.542985 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:09.543219 (+ 234us) server_negotiation.cc:299] Negotiation successful
0807 07:02:09.543447 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260807 07:02:09.548739 30960 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:09.547323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36716 (local address 127.21.83.1:46431)
0807 07:02:09.547500 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:09.547506 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:09.547740 (+ 234us) server_negotiation.cc:407] Connection header received
0807 07:02:09.547849 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:09.547852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:09.547903 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:09.547974 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:09.548541 (+ 567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36716: BlockingRecv error: recv got EOF from 127.0.0.1:36716 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260807 07:02:09.549185 21836 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/krb5kdc/krb5cc)
Aug 07 07:02:09 dist-test-slave-ncp9 krb5kdc[30246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:02:09.550592 21836 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 07 07:02:09 dist-test-slave-ncp9 krb5kdc[30246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:02:09.551755 21836 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 07 07:02:09 dist-test-slave-ncp9 krb5kdc[30246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.1@KRBTEST.COM
I20260807 07:02:09.563911 30960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:09.552945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36722 (local address 127.21.83.1:46431)
0807 07:02:09.553104 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:09.553109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:09.553265 (+ 156us) server_negotiation.cc:407] Connection header received
0807 07:02:09.553366 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:09.553369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:09.553416 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:09.553485 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:09.554332 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:09.554982 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:09.556087 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:09.556472 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:09.556563 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:09.559568 (+ 3005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:09.559573 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:09.559576 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:09.559613 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:09.561877 (+ 2264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:09.562391 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:09.562394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:09.562396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:09.562500 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:09.562760 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:09.562763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:09.562765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:09.562989 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:09.563249 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:09.563503 (+ 254us) server_negotiation.cc:299] Negotiation successful
0807 07:02:09.563654 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260807 07:02:09.567592 30960 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:09.566234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36728 (local address 127.21.83.1:46431)
0807 07:02:09.566408 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:09.566413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:09.566602 (+ 189us) server_negotiation.cc:407] Connection header received
0807 07:02:09.566707 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:09.566710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:09.566772 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:09.566841 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:09.567397 (+ 556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36728: BlockingRecv error: recv got EOF from 127.0.0.1:36728 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260807 07:02:09.567770 21836 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786086078080656-21836-0/krb5kdc/krb5cc' of type FILE
Aug 07 07:02:09 dist-test-slave-ncp9 krb5kdc[30246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:02:09.569180 21836 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 07 07:02:09 dist-test-slave-ncp9 krb5kdc[30246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:02:09.570343 21836 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 07 07:02:09 dist-test-slave-ncp9 krb5kdc[30246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.1@KRBTEST.COM
I20260807 07:02:09.582239 30960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:09.571524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36740 (local address 127.21.83.1:46431)
0807 07:02:09.571741 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:09.571747 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:09.571898 (+ 151us) server_negotiation.cc:407] Connection header received
0807 07:02:09.572005 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:09.572008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:09.572056 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:09.572120 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:09.573009 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:09.573671 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:09.574777 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:09.575145 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:09.575227 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:09.578141 (+ 2914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:09.578145 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:09.578148 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:09.578175 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:09.580278 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:09.580782 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:09.580785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:09.580787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:09.580927 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:09.581215 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:09.581217 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:09.581219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:09.581436 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:09.581630 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:09.581786 (+ 156us) server_negotiation.cc:299] Negotiation successful
0807 07:02:09.581944 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260807 07:02:09.583648 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 30390
I20260807 07:02:09.596933 30960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:09.584313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36742 (local address 127.21.83.1:46431)
0807 07:02:09.584680 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:09.584686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:09.584862 (+ 176us) server_negotiation.cc:407] Connection header received
0807 07:02:09.585005 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:09.585009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:09.585091 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:09.585214 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:09.586136 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:09.586843 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:09.588336 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:09.588796 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:09.588948 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:09.591620 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:09.591626 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:09.591629 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:09.591673 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:09.594354 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:09.595158 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:09.595163 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:09.595165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:09.595319 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:09.595620 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:09.595623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:09.595625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:09.595890 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:09.596110 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:09.596418 (+ 308us) server_negotiation.cc:299] Negotiation successful
0807 07:02:09.596620 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260807 07:02:09.597553 30558 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:02:09.684449 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 30390
I20260807 07:02:09.707799 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 30577
Aug 07 07:02:09 dist-test-slave-ncp9 krb5kdc[30246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.2@KRBTEST.COM
I20260807 07:02:09.723011 30976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:09.708645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41142 (local address 127.21.83.2:46059)
0807 07:02:09.709173 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:09.709180 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:09.709207 (+ 27us) server_negotiation.cc:407] Connection header received
0807 07:02:09.709270 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:09.709277 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:09.709481 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:09.709609 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:09.710492 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:09.711475 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:09.712643 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:09.713343 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:09.713446 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:09.717145 (+ 3699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:09.717161 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:09.717179 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:09.717222 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:09.720062 (+ 2840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:09.721017 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:09.721026 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:09.721028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:09.721156 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:09.721461 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:09.721464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:09.721466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:09.721832 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:09.722074 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:09.722420 (+ 346us) server_negotiation.cc:299] Negotiation successful
0807 07:02:09.722656 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":271,"threads_started":1}
I20260807 07:02:09.724303 30756 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:02:09.808106 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 30577
I20260807 07:02:09.832496 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 30772
Aug 07 07:02:09 dist-test-slave-ncp9 krb5kdc[30246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.3@KRBTEST.COM
I20260807 07:02:09.847057 30978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:09.833436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54278 (local address 127.21.83.3:36141)
0807 07:02:09.834002 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:09.834009 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:09.834053 (+ 44us) server_negotiation.cc:407] Connection header received
0807 07:02:09.834116 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:09.834122 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:09.834333 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:09.834460 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:09.835397 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:09.836410 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:09.837623 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:09.838161 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:09.838255 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:09.842025 (+ 3770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:09.842043 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:09.842057 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:09.842090 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:09.844574 (+ 2484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:09.845240 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:09.845247 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:09.845249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:09.845367 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:09.845655 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:09.845658 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:09.845660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:09.845982 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:09.846214 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:09.846488 (+ 274us) server_negotiation.cc:299] Negotiation successful
0807 07:02:09.846772 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":279,"threads_started":1}
I20260807 07:02:09.848493 30941 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:02:09.932265 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 30772
I20260807 07:02:09.953833 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 30263
I20260807 07:02:09.955116 30347 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:02:10.065822 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 30263
2026-08-07T07:02:10Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4124 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 07:02:10 dist-test-slave-ncp9 krb5kdc[30984](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 07:02:10 dist-test-slave-ncp9 krb5kdc[30984](info): set up 2 sockets
Aug 07 07:02:10 dist-test-slave-ncp9 krb5kdc[30984](info): commencing operation
krb5kdc: starting...
W20260807 07:02:11.272157 21836 mini_kdc.cc:121] Time spent starting KDC: real 1.159s user 0.001s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Aug 07 07:02:11 dist-test-slave-ncp9 krb5kdc[30984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T07:02:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:02:11Z Disabled control of system clock
WARNING: no policy specified for oryx/127.21.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.62.keytab.
Entry for principal oryx/127.21.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.62.keytab.
I20260807 07:02:11.530300 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:45341
--webserver_interface=127.21.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.62.keytab
--principal=oryx/127.21.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:41345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:45341
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:02:11.870626 31004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:02:11.870985 31004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:02:11.871119 31004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:02:11.878973 31004 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:02:11.879120 31004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:02:11.879184 31004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:02:11.879267 31004 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:02:11.879325 31004 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:02:11.889932 31004 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:41345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:45341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.62.keytab
--principal=oryx/127.21.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:45341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:02:11.891569 31004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:02:11.893050 31004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:02:11.905694 31013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:11.905858 31016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:11.906054 31012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:02:11.907953 31004 server_base.cc:1034] running on GCE node
I20260807 07:02:11.908744 31004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:02:11.910198 31004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:02:11.911406 31004 hybrid_clock.cc:648] HybridClock initialized: now 1786086131911348 us; error 48 us; skew 500 ppm
I20260807 07:02:11.911746 31004 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:02:11 dist-test-slave-ncp9 krb5kdc[30984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086131, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:02:11.914798 31004 init.cc:377] Logged in from keytab as oryx/127.21.83.62@KRBTEST.COM (short username oryx)
I20260807 07:02:11.916229 31004 webserver.cc:460] Webserver started at http://127.21.83.62:38481/ using document root <none> and password file <none>
I20260807 07:02:11.916932 31004 fs_manager.cc:359] Metadata directory not provided
I20260807 07:02:11.917073 31004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:02:11.917371 31004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:02:11.920317 31004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "9c117a2eae7c43dd92d88685ac08f1eb"
format_stamp: "Formatted at 2026-08-07 07:02:11 on dist-test-slave-ncp9"
server_key: "b13cf1923d0ea7912be7e26eb9f4051a"
server_key_iv: "fca1326776e8aea33f01e29b904a2e11"
server_key_version: "encryptionkey@0"
I20260807 07:02:11.921124 31004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "9c117a2eae7c43dd92d88685ac08f1eb"
format_stamp: "Formatted at 2026-08-07 07:02:11 on dist-test-slave-ncp9"
server_key: "b13cf1923d0ea7912be7e26eb9f4051a"
server_key_iv: "fca1326776e8aea33f01e29b904a2e11"
server_key_version: "encryptionkey@0"
I20260807 07:02:11.926227 31004 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260807 07:02:11.929564 31026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:11.930775 31004 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 07:02:11.930936 31004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "9c117a2eae7c43dd92d88685ac08f1eb"
format_stamp: "Formatted at 2026-08-07 07:02:11 on dist-test-slave-ncp9"
server_key: "b13cf1923d0ea7912be7e26eb9f4051a"
server_key_iv: "fca1326776e8aea33f01e29b904a2e11"
server_key_version: "encryptionkey@0"
I20260807 07:02:11.931114 31004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:02:11.947126 31004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:02:11.951532 31004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:02:11.951805 31004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:02:11.970080 31004 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.62:45341
I20260807 07:02:11.970112 31100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.62:45341 every 8 connection(s)
I20260807 07:02:11.971374 31004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:02:11.975238 31102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:02:11.977047 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 31004
I20260807 07:02:11.977296 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:02:11.978120 21836 external_mini_cluster.cc:1428] Setting key 9b16dbb817248dbb01cdc84493de2f30
I20260807 07:02:11.989156 31102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb: Bootstrap starting.
I20260807 07:02:11.992588 31102 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb: Neither blocks nor log segments found. Creating new log.
I20260807 07:02:11.993729 31102 log.cc:824] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb: Log is configured to *not* fsync() on all Append() calls
Aug 07 07:02:11 dist-test-slave-ncp9 krb5kdc[30984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM
I20260807 07:02:11.997218 31102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb: No bootstrap required, opened a new log
I20260807 07:02:12.005069 31105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:11.981179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58986 (local address 127.21.83.62:45341)
0807 07:02:11.982342 (+ 1163us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:11.982356 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:11.982414 (+ 58us) server_negotiation.cc:407] Connection header received
0807 07:02:11.984733 (+ 2319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:11.984775 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:11.985602 (+ 827us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:11.986150 (+ 548us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:11.988071 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:11.989629 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:11.991013 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:11.991911 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:11.992127 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:11.996806 (+ 4679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:11.996816 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:11.996832 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:11.996880 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:12.000135 (+ 3255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:12.000983 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:12.000991 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:12.000998 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:12.001196 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:12.001624 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:12.001628 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:12.001630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:12.002304 (+ 674us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:12.002669 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:12.003141 (+ 472us) server_negotiation.cc:299] Negotiation successful
0807 07:02:12.003732 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":863,"thread_start_us":572,"threads_started":1}
I20260807 07:02:12.006033 31102 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } }
I20260807 07:02:12.006729 31102 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:02:12.006878 31102 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c117a2eae7c43dd92d88685ac08f1eb, State: Initialized, Role: FOLLOWER
I20260807 07:02:12.007596 31102 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } }
I20260807 07:02:12.007882 31102 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:02:12.008040 31102 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:02:12.008251 31102 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:02:12.010639 31102 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } }
I20260807 07:02:12.011178 31102 leader_election.cc:302] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c117a2eae7c43dd92d88685ac08f1eb; no voters:
I20260807 07:02:12.011786 31102 leader_election.cc:288] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 07:02:12.011940 31109 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:02:12.012607 31109 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb [term 1 LEADER]: Becoming Leader. State: Replica: 9c117a2eae7c43dd92d88685ac08f1eb, State: Running, Role: LEADER
I20260807 07:02:12.013228 31109 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } }
I20260807 07:02:12.014108 31102 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:02:12.016301 31111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } }
I20260807 07:02:12.016273 31112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c117a2eae7c43dd92d88685ac08f1eb. Latest consensus state: current_term: 1 leader_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } }
I20260807 07:02:12.016834 31112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb [sys.catalog]: This master's current role is: LEADER
I20260807 07:02:12.016834 31111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb [sys.catalog]: This master's current role is: LEADER
I20260807 07:02:12.018967 31119 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:02:12.023826 31119 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:02:12.032725 31119 catalog_manager.cc:1347] Generated new cluster ID: b2a37eafb25945a195ac09feead69fd5
I20260807 07:02:12.032928 31119 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:02:12.052090 31119 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:02:12.053632 31119 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:02:12.062836 31119 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb: Generated new TSK 0
I20260807 07:02:12.063696 31119 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.21.83.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.83.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.83.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.1.keytab.
Entry for principal oryx/127.21.83.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.1.keytab.
I20260807 07:02:12.130203 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.1:0
--local_ip_for_outbound_sockets=127.21.83.1
--webserver_interface=127.21.83.1
--webserver_port=0
--tserver_master_addrs=127.21.83.62:45341
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.1.keytab
--principal=oryx/127.21.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:41345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:02:12.459789 31140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:02:12.460036 31140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:02:12.460157 31140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:02:12.467108 31140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:02:12.467204 31140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:02:12.467341 31140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.1
I20260807 07:02:12.476467 31140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:41345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.1.keytab
--principal=oryx/127.21.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.83.1
--webserver_port=0
--tserver_master_addrs=127.21.83.62:45341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.1
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:02:12.477628 31140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:02:12.478873 31140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:02:12.490811 31148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:12.490907 31147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:12.491114 31152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:02:12.491901 31140 server_base.cc:1034] running on GCE node
I20260807 07:02:12.492416 31140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:02:12.493259 31140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:02:12.494449 31140 hybrid_clock.cc:648] HybridClock initialized: now 1786086132494407 us; error 35 us; skew 500 ppm
I20260807 07:02:12.494724 31140 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:02:12 dist-test-slave-ncp9 krb5kdc[30984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086132, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:02:12.497738 31140 init.cc:377] Logged in from keytab as oryx/127.21.83.1@KRBTEST.COM (short username oryx)
I20260807 07:02:12.499006 31140 webserver.cc:460] Webserver started at http://127.21.83.1:45839/ using document root <none> and password file <none>
I20260807 07:02:12.499615 31140 fs_manager.cc:359] Metadata directory not provided
I20260807 07:02:12.499735 31140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:02:12.500017 31140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:02:12.503036 31140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-0/data/instance:
uuid: "12c53d5fb1164a938c89f771a0d186d4"
format_stamp: "Formatted at 2026-08-07 07:02:12 on dist-test-slave-ncp9"
server_key: "a1934cabac7994b3550b5e0004618cb9"
server_key_iv: "6bea95e64fce2b21939fdd75c8700b57"
server_key_version: "encryptionkey@0"
I20260807 07:02:12.503911 31140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-0/wal/instance:
uuid: "12c53d5fb1164a938c89f771a0d186d4"
format_stamp: "Formatted at 2026-08-07 07:02:12 on dist-test-slave-ncp9"
server_key: "a1934cabac7994b3550b5e0004618cb9"
server_key_iv: "6bea95e64fce2b21939fdd75c8700b57"
server_key_version: "encryptionkey@0"
I20260807 07:02:12.509903 31140 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260807 07:02:12.513923 31161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:12.515558 31140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260807 07:02:12.515774 31140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-0/wal
uuid: "12c53d5fb1164a938c89f771a0d186d4"
format_stamp: "Formatted at 2026-08-07 07:02:12 on dist-test-slave-ncp9"
server_key: "a1934cabac7994b3550b5e0004618cb9"
server_key_iv: "6bea95e64fce2b21939fdd75c8700b57"
server_key_version: "encryptionkey@0"
I20260807 07:02:12.516003 31140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:02:12.537942 31140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:02:12.542654 31140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:02:12.542965 31140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:02:12.544056 31140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:02:12.545588 31140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:02:12.545668 31140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:12.545794 31140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:02:12.545848 31140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:12.578853 31140 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.1:39755
I20260807 07:02:12.578900 31324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.1:39755 every 8 connection(s)
I20260807 07:02:12.580211 31140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
I20260807 07:02:12.584877 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 31140
I20260807 07:02:12.585119 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-0/wal/instance
I20260807 07:02:12.585754 21836 external_mini_cluster.cc:1428] Setting key 8bb966818653be997f21742a2e4ba693
Aug 07 07:02:12 dist-test-slave-ncp9 krb5kdc[30984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086132, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.83.1@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM
I20260807 07:02:12.602252 31328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:12.582934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:49895 (local address 127.21.83.62:45341)
0807 07:02:12.583430 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:12.583435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:12.584503 (+ 1068us) server_negotiation.cc:407] Connection header received
0807 07:02:12.586694 (+ 2191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:12.586701 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:12.586798 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:12.586917 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:12.588728 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:12.589501 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:12.591134 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:12.591543 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:12.591675 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:12.595407 (+ 3732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:12.595413 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:12.595416 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:12.595454 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:12.598049 (+ 2595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:12.598857 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:12.598862 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:12.598865 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:12.599011 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:12.599445 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:12.599448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:12.599450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:12.599739 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:12.599877 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:12.601234 (+ 1357us) server_negotiation.cc:299] Negotiation successful
0807 07:02:12.601632 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":241,"threads_started":1}
I20260807 07:02:12.603544 31329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:12.583509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:45341 (local address 127.21.83.1:49895)
0807 07:02:12.584283 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:12.584332 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:12.586294 (+ 1962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:12.587211 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:12.587223 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:12.587882 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:12.588543 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:12.588562 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:12.589665 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:12.589669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:12.590966 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:12.590980 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:12.591802 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:12.591809 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:12.592044 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:12.592772 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:12.592801 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:12.595162 (+ 2361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:12.598218 (+ 3056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:12.598228 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:12.598249 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:12.598730 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:12.599123 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:12.599127 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:12.599131 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:12.599338 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:12.599868 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:12.599877 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:12.600252 (+ 375us) client_negotiation.cc:769] Sending connection context
0807 07:02:12.601020 (+ 768us) client_negotiation.cc:240] Negotiation successful
0807 07:02:12.602066 (+ 1046us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":303,"threads_started":1}
I20260807 07:02:12.605360 31325 heartbeater.cc:347] Connected to a master server at 127.21.83.62:45341
I20260807 07:02:12.605719 31325 heartbeater.cc:464] Registering TS with master...
I20260807 07:02:12.606668 31325 heartbeater.cc:511] Master 127.21.83.62:45341 requested a full tablet report, sending...
I20260807 07:02:12.609426 31048 ts_manager.cc:194] Registered new tserver with Master: 12c53d5fb1164a938c89f771a0d186d4 (127.21.83.1:39755)
I20260807 07:02:12.611572 31048 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.83.1@KRBTEST.COM'} at 127.21.83.1:49895
WARNING: no policy specified for oryx/127.21.83.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.83.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.83.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.2.keytab.
Entry for principal oryx/127.21.83.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.2.keytab.
I20260807 07:02:12.646219 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.2:0
--local_ip_for_outbound_sockets=127.21.83.2
--webserver_interface=127.21.83.2
--webserver_port=0
--tserver_master_addrs=127.21.83.62:45341
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.2.keytab
--principal=oryx/127.21.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:41345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:02:13.001061 31335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:02:13.001384 31335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:02:13.001528 31335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:02:13.008822 31335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:02:13.008972 31335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:02:13.009132 31335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.2
I20260807 07:02:13.018991 31335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:41345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.2.keytab
--principal=oryx/127.21.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.83.2
--webserver_port=0
--tserver_master_addrs=127.21.83.62:45341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.2
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:02:13.020252 31335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:02:13.021667 31335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:02:13.034075 31340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:13.034065 31342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:13.034303 31344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:02:13.035023 31335 server_base.cc:1034] running on GCE node
I20260807 07:02:13.035606 31335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:02:13.036473 31335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:02:13.037660 31335 hybrid_clock.cc:648] HybridClock initialized: now 1786086133037614 us; error 36 us; skew 500 ppm
I20260807 07:02:13.037932 31335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:02:13 dist-test-slave-ncp9 krb5kdc[30984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086133, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:02:13.040813 31335 init.cc:377] Logged in from keytab as oryx/127.21.83.2@KRBTEST.COM (short username oryx)
I20260807 07:02:13.042210 31335 webserver.cc:460] Webserver started at http://127.21.83.2:33513/ using document root <none> and password file <none>
I20260807 07:02:13.042809 31335 fs_manager.cc:359] Metadata directory not provided
I20260807 07:02:13.042909 31335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:02:13.043164 31335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:02:13.046044 31335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-1/data/instance:
uuid: "369da2bd392a4341ba2c5af643fc6e84"
format_stamp: "Formatted at 2026-08-07 07:02:13 on dist-test-slave-ncp9"
server_key: "000a5a64ef2d915509f3987156f0da8a"
server_key_iv: "891a28d01dc95d15cb7fb0d7d99059f5"
server_key_version: "encryptionkey@0"
I20260807 07:02:13.046787 31335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-1/wal/instance:
uuid: "369da2bd392a4341ba2c5af643fc6e84"
format_stamp: "Formatted at 2026-08-07 07:02:13 on dist-test-slave-ncp9"
server_key: "000a5a64ef2d915509f3987156f0da8a"
server_key_iv: "891a28d01dc95d15cb7fb0d7d99059f5"
server_key_version: "encryptionkey@0"
I20260807 07:02:13.051740 31335 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 07:02:13.055121 31353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:13.056420 31335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 07:02:13.056550 31335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-1/wal
uuid: "369da2bd392a4341ba2c5af643fc6e84"
format_stamp: "Formatted at 2026-08-07 07:02:13 on dist-test-slave-ncp9"
server_key: "000a5a64ef2d915509f3987156f0da8a"
server_key_iv: "891a28d01dc95d15cb7fb0d7d99059f5"
server_key_version: "encryptionkey@0"
I20260807 07:02:13.056711 31335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:02:13.073200 31335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:02:13.076620 31335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:02:13.076843 31335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:02:13.077739 31335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:02:13.078975 31335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:02:13.079051 31335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:13.079146 31335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:02:13.079190 31335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:13.110910 31335 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.2:45967
I20260807 07:02:13.110954 31513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.2:45967 every 8 connection(s)
I20260807 07:02:13.112323 31335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
I20260807 07:02:13.113232 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 31335
I20260807 07:02:13.113435 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-1/wal/instance
I20260807 07:02:13.114046 21836 external_mini_cluster.cc:1428] Setting key 2a20704ec507bb7f23d9b25b7cdaf0a0
Aug 07 07:02:13 dist-test-slave-ncp9 krb5kdc[30984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086133, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.83.2@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM
I20260807 07:02:13.135128 31518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:13.115506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:38289 (local address 127.21.83.62:45341)
0807 07:02:13.116022 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:13.116026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:13.117313 (+ 1287us) server_negotiation.cc:407] Connection header received
0807 07:02:13.119533 (+ 2220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:13.119540 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:13.119659 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:13.119762 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:13.121767 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:13.122557 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:13.124240 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:13.124676 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:13.124799 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:13.128815 (+ 4016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:13.128821 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:13.128824 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:13.128863 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:13.131553 (+ 2690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:13.132432 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:13.132437 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:13.132440 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:13.132579 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:13.133113 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:13.133117 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:13.133119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:13.133410 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:13.133555 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:13.134383 (+ 828us) server_negotiation.cc:299] Negotiation successful
0807 07:02:13.134647 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":274,"threads_started":1}
I20260807 07:02:13.136157 31519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:13.116282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:45341 (local address 127.21.83.2:38289)
0807 07:02:13.117156 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:13.117206 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:13.119219 (+ 2013us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:13.120109 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:13.120119 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:13.120808 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:13.121547 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:13.121572 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:13.122743 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:13.122748 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:13.124065 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:13.124082 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:13.124972 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:13.124981 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:13.125228 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:13.125968 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:13.125999 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:13.128520 (+ 2521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:13.131743 (+ 3223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:13.131754 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:13.131773 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:13.132287 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:13.132701 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:13.132707 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:13.132711 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:13.132988 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:13.133565 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:13.133574 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:13.133972 (+ 398us) client_negotiation.cc:769] Sending connection context
0807 07:02:13.134266 (+ 294us) client_negotiation.cc:240] Negotiation successful
0807 07:02:13.134866 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":353,"threads_started":1}
I20260807 07:02:13.137789 31516 heartbeater.cc:347] Connected to a master server at 127.21.83.62:45341
I20260807 07:02:13.138190 31516 heartbeater.cc:464] Registering TS with master...
I20260807 07:02:13.139375 31516 heartbeater.cc:511] Master 127.21.83.62:45341 requested a full tablet report, sending...
I20260807 07:02:13.141336 31048 ts_manager.cc:194] Registered new tserver with Master: 369da2bd392a4341ba2c5af643fc6e84 (127.21.83.2:45967)
WARNING: no policy specified for oryx/127.21.83.3@KRBTEST.COM; defaulting to no policy
I20260807 07:02:13.142217 31048 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.83.2@KRBTEST.COM'} at 127.21.83.2:38289
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.83.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.83.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.3.keytab.
Entry for principal oryx/127.21.83.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.3.keytab.
I20260807 07:02:13.175387 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.3:0
--local_ip_for_outbound_sockets=127.21.83.3
--webserver_interface=127.21.83.3
--webserver_port=0
--tserver_master_addrs=127.21.83.62:45341
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.3.keytab
--principal=oryx/127.21.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:41345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:02:13.516004 31523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:02:13.516274 31523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:02:13.516430 31523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:02:13.524098 31523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:02:13.524216 31523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:02:13.524395 31523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.3
I20260807 07:02:13.534369 31523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:41345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.3.keytab
--principal=oryx/127.21.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.83.3
--webserver_port=0
--tserver_master_addrs=127.21.83.62:45341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.3
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:02:13.535657 31523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:02:13.537087 31523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:02:13.549889 31531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:13.550067 31532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:13.550096 31535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:02:13.551007 31523 server_base.cc:1034] running on GCE node
I20260807 07:02:13.551587 31523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:02:13.552538 31523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:02:13.553740 31523 hybrid_clock.cc:648] HybridClock initialized: now 1786086133553696 us; error 35 us; skew 500 ppm
I20260807 07:02:13.554041 31523 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:02:13 dist-test-slave-ncp9 krb5kdc[30984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086133, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:02:13.561015 31523 init.cc:377] Logged in from keytab as oryx/127.21.83.3@KRBTEST.COM (short username oryx)
I20260807 07:02:13.562559 31523 webserver.cc:460] Webserver started at http://127.21.83.3:37255/ using document root <none> and password file <none>
I20260807 07:02:13.563256 31523 fs_manager.cc:359] Metadata directory not provided
I20260807 07:02:13.563395 31523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:02:13.563715 31523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:02:13.566854 31523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-2/data/instance:
uuid: "04c5ff42c0024a9e8f9014904318804e"
format_stamp: "Formatted at 2026-08-07 07:02:13 on dist-test-slave-ncp9"
server_key: "42c9009fb40c4f0b1a7f819855cbce88"
server_key_iv: "1142d1855c5310dd44f404a158a3b428"
server_key_version: "encryptionkey@0"
I20260807 07:02:13.567694 31523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-2/wal/instance:
uuid: "04c5ff42c0024a9e8f9014904318804e"
format_stamp: "Formatted at 2026-08-07 07:02:13 on dist-test-slave-ncp9"
server_key: "42c9009fb40c4f0b1a7f819855cbce88"
server_key_iv: "1142d1855c5310dd44f404a158a3b428"
server_key_version: "encryptionkey@0"
I20260807 07:02:13.572975 31523 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260807 07:02:13.576239 31542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:13.577466 31523 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 07:02:13.577618 31523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-2/wal
uuid: "04c5ff42c0024a9e8f9014904318804e"
format_stamp: "Formatted at 2026-08-07 07:02:13 on dist-test-slave-ncp9"
server_key: "42c9009fb40c4f0b1a7f819855cbce88"
server_key_iv: "1142d1855c5310dd44f404a158a3b428"
server_key_version: "encryptionkey@0"
I20260807 07:02:13.577791 31523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:02:13.609701 31523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:02:13.613663 31523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:02:13.613929 31523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:02:13.614985 31523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:02:13.615213 31325 heartbeater.cc:503] Master 127.21.83.62:45341 was elected leader, sending a full tablet report...
I20260807 07:02:13.616452 31523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:02:13.616534 31523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:13.616662 31523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:02:13.616714 31523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:13.648826 31523 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.3:42015
I20260807 07:02:13.648875 31710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.3:42015 every 8 connection(s)
I20260807 07:02:13.650120 31523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
I20260807 07:02:13.653131 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 31523
I20260807 07:02:13.653285 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-2/wal/instance
I20260807 07:02:13.653851 21836 external_mini_cluster.cc:1428] Setting key 68e32ab59e2665213055abb27fe1e4a2
Aug 07 07:02:13 dist-test-slave-ncp9 krb5kdc[30984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086133, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.83.3@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM
I20260807 07:02:13.670006 31715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:13.652568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:34945 (local address 127.21.83.62:45341)
0807 07:02:13.653058 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:13.653064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:13.654121 (+ 1057us) server_negotiation.cc:407] Connection header received
0807 07:02:13.656404 (+ 2283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:13.656410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:13.656498 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:13.656589 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:13.658160 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:13.658930 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:13.660564 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:13.661052 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:13.661158 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:13.664506 (+ 3348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:13.664511 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:13.664514 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:13.664547 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:13.667037 (+ 2490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:13.667696 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:13.667702 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:13.667706 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:13.667824 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:13.668237 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:13.668242 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:13.668245 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:13.668528 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:13.668680 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:13.669400 (+ 720us) server_negotiation.cc:299] Negotiation successful
0807 07:02:13.669633 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":211,"threads_started":1}
I20260807 07:02:13.670876 31717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:13.653127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:45341 (local address 127.21.83.3:34945)
0807 07:02:13.653934 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:13.654010 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:13.656136 (+ 2126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:13.656915 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:13.656929 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:13.657451 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:13.657998 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:13.658016 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:13.659098 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:13.659103 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:13.660428 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:13.660441 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:13.661285 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:13.661293 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:13.661498 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:13.662118 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:13.662145 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:13.664295 (+ 2150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:13.667164 (+ 2869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:13.667173 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:13.667187 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:13.667585 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:13.667928 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:13.667933 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:13.667937 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:13.668129 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:13.668655 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:13.668664 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:13.669045 (+ 381us) client_negotiation.cc:769] Sending connection context
0807 07:02:13.669293 (+ 248us) client_negotiation.cc:240] Negotiation successful
0807 07:02:13.669798 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":321,"threads_started":1}
I20260807 07:02:13.672075 31711 heartbeater.cc:347] Connected to a master server at 127.21.83.62:45341
I20260807 07:02:13.672334 31711 heartbeater.cc:464] Registering TS with master...
I20260807 07:02:13.673070 31711 heartbeater.cc:511] Master 127.21.83.62:45341 requested a full tablet report, sending...
I20260807 07:02:13.674537 31048 ts_manager.cc:194] Registered new tserver with Master: 04c5ff42c0024a9e8f9014904318804e (127.21.83.3:42015)
I20260807 07:02:13.675280 31048 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.83.3@KRBTEST.COM'} at 127.21.83.3:34945
I20260807 07:02:13.681020 21836 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.21.83.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.83.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.83.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.61.keytab.
Entry for principal oryx/127.21.83.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.61.keytab.
I20260807 07:02:13.737584 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.61:34401
--webserver_interface=127.21.83.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.61.keytab
--principal=oryx/127.21.83.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:41345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:45341,127.21.83.61:34401
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:02:14.064253 31721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:02:14.064541 31721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:02:14.064674 31721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:02:14.071954 31721 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:02:14.072082 31721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:02:14.072163 31721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:02:14.072243 31721 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:02:14.072304 31721 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:02:14.082743 31721 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:41345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:45341,127.21.83.61:34401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.61.keytab
--principal=oryx/127.21.83.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.61:34401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.83.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
W20260807 07:02:14.083112 31721 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.21.83.62:45341,127.21.83.61:34401'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260807 07:02:14.084127 31721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:02:14.085479 31721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:02:14.096629 31731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:14.096557 31729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:14.096643 31728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:02:14.097515 31721 server_base.cc:1034] running on GCE node
I20260807 07:02:14.098207 31721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:02:14.099483 31721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:02:14.100687 31721 hybrid_clock.cc:648] HybridClock initialized: now 1786086134100628 us; error 34 us; skew 500 ppm
I20260807 07:02:14.101042 31721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:02:14 dist-test-slave-ncp9 krb5kdc[30984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086134, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.83.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:02:14.103992 31721 init.cc:377] Logged in from keytab as oryx/127.21.83.61@KRBTEST.COM (short username oryx)
I20260807 07:02:14.105460 31721 webserver.cc:460] Webserver started at http://127.21.83.61:38339/ using document root <none> and password file <none>
I20260807 07:02:14.106083 31721 fs_manager.cc:359] Metadata directory not provided
I20260807 07:02:14.106166 31721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:02:14.106410 31721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:02:14.109187 31721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-1/data/instance:
uuid: "a0998212e8024827a4228dd4ca6ed392"
format_stamp: "Formatted at 2026-08-07 07:02:14 on dist-test-slave-ncp9"
server_key: "197c164793913b7db3b2b80d1dd99136"
server_key_iv: "34dd029e44952922f49e4d01d11670d2"
server_key_version: "encryptionkey@0"
I20260807 07:02:14.109887 31721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-1/wal/instance:
uuid: "a0998212e8024827a4228dd4ca6ed392"
format_stamp: "Formatted at 2026-08-07 07:02:14 on dist-test-slave-ncp9"
server_key: "197c164793913b7db3b2b80d1dd99136"
server_key_iv: "34dd029e44952922f49e4d01d11670d2"
server_key_version: "encryptionkey@0"
I20260807 07:02:14.116149 31721 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260807 07:02:14.119381 31738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:14.120601 31721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260807 07:02:14.120756 31721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-1/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-1/wal
uuid: "a0998212e8024827a4228dd4ca6ed392"
format_stamp: "Formatted at 2026-08-07 07:02:14 on dist-test-slave-ncp9"
server_key: "197c164793913b7db3b2b80d1dd99136"
server_key_iv: "34dd029e44952922f49e4d01d11670d2"
server_key_version: "encryptionkey@0"
I20260807 07:02:14.120966 31721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:02:14.144378 31721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:02:14.145318 31516 heartbeater.cc:503] Master 127.21.83.62:45341 was elected leader, sending a full tablet report...
I20260807 07:02:14.149492 31721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:02:14.149734 31721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:02:14.166778 31721 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.61:34401
I20260807 07:02:14.166790 31818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.61:34401 every 8 connection(s)
I20260807 07:02:14.168054 31721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-1/data/info.pb
I20260807 07:02:14.169893 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 31721
I20260807 07:02:14.172585 31819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:02:14.181507 31819 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 07:02:14 dist-test-slave-ncp9 krb5kdc[30984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.83.61@KRBTEST.COM
Aug 07 07:02:14 dist-test-slave-ncp9 krb5kdc[30984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086134, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.83.61@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM
I20260807 07:02:14.194188 31823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:14.172172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49096 (local address 127.21.83.61:34401)
0807 07:02:14.173126 (+ 954us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:14.173142 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:14.173763 (+ 621us) server_negotiation.cc:407] Connection header received
0807 07:02:14.175613 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:14.175641 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:14.176120 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:14.176553 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:14.177937 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:14.179894 (+ 1957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:14.181183 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:14.181847 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:14.182164 (+ 317us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:14.186687 (+ 4523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:14.186697 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:14.186709 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:14.186758 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:14.190040 (+ 3282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:14.190805 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:14.190811 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:14.190817 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:14.191024 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:14.191418 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:14.191423 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:14.191426 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:14.192099 (+ 673us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:14.192382 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:14.192778 (+ 396us) server_negotiation.cc:299] Negotiation successful
0807 07:02:14.193197 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":382,"threads_started":1}
I20260807 07:02:14.199539 31824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:14.184276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58996 (local address 127.21.83.62:45341)
0807 07:02:14.184769 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:14.184775 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:14.185317 (+ 542us) server_negotiation.cc:407] Connection header received
0807 07:02:14.185489 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:14.185495 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:14.185605 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:14.185720 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:14.186824 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:14.187778 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:14.189774 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:14.190234 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:14.190374 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:14.193752 (+ 3378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:14.193758 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:14.193761 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:14.193798 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:14.196957 (+ 3159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:14.197786 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:14.197790 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:14.197793 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:14.197927 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:14.198231 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:14.198234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:14.198236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:14.198567 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:14.198695 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:14.199084 (+ 389us) server_negotiation.cc:299] Negotiation successful
0807 07:02:14.199290 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":246,"threads_started":1}
I20260807 07:02:14.199570 31825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:14.184224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:45341 (local address 127.0.0.1:58996)
0807 07:02:14.185176 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:14.185224 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:14.185367 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:14.185770 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:14.185776 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:14.186348 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:14.186675 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:14.186687 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:14.187934 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:14.187939 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:14.189618 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:14.189645 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:14.190401 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:14.190405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:14.190525 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:14.191028 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:14.191050 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:14.193565 (+ 2515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:14.197103 (+ 3538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:14.197114 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:14.197120 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:14.197649 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:14.198010 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:14.198013 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:14.198015 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:14.198136 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:14.198665 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:14.198671 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:14.198840 (+ 169us) client_negotiation.cc:769] Sending connection context
0807 07:02:14.198998 (+ 158us) client_negotiation.cc:240] Negotiation successful
0807 07:02:14.199271 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"thread_start_us":562,"threads_started":1}
I20260807 07:02:14.201311 31819 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.83.61" port: 34401 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 07:02:14 dist-test-slave-ncp9 krb5kdc[30984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086134, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.83.61@KRBTEST.COM for oryx/127.21.83.61@KRBTEST.COM
I20260807 07:02:14.213652 31825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:14.201983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.61:34401 (local address 127.0.0.1:49098)
0807 07:02:14.202158 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:14.202174 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:14.202283 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:14.202516 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:14.202519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:14.202879 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:14.203023 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:14.203032 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:14.204006 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:14.204011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:14.204959 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:14.204974 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:14.205646 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:14.205650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:14.205773 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:14.206333 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:14.206350 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:14.208433 (+ 2083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:14.211444 (+ 3011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:14.211448 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:14.211451 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:14.211857 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:14.212214 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:14.212218 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:14.212220 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:14.212349 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:14.212842 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:14.212846 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:14.213019 (+ 173us) client_negotiation.cc:769] Sending connection context
0807 07:02:14.213155 (+ 136us) client_negotiation.cc:240] Negotiation successful
0807 07:02:14.213390 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260807 07:02:14.213918 31823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:14.202036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49098 (local address 127.21.83.61:34401)
0807 07:02:14.202181 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:14.202186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:14.202228 (+ 42us) server_negotiation.cc:407] Connection header received
0807 07:02:14.202355 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:14.202358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:14.202413 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:14.202492 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:14.203138 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:14.203846 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:14.205089 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:14.205494 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:14.205587 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:14.208620 (+ 3033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:14.208634 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:14.208638 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:14.208669 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:14.211286 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:14.211960 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:14.211964 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:14.211967 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:14.212110 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:14.212440 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:14.212444 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:14.212445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:14.212740 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:14.212852 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:14.213243 (+ 391us) server_negotiation.cc:299] Negotiation successful
0807 07:02:14.213559 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":86,"server-negotiator.queue_time_us":48}
I20260807 07:02:14.218803 31819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392: Bootstrap starting.
I20260807 07:02:14.222424 31819 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392: Neither blocks nor log segments found. Creating new log.
I20260807 07:02:14.223521 31819 log.cc:824] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392: Log is configured to *not* fsync() on all Append() calls
I20260807 07:02:14.226488 31819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392: No bootstrap required, opened a new log
I20260807 07:02:14.233700 31819 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } peers { permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 34401 } }
I20260807 07:02:14.234048 31819 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:02:14.234182 31819 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0998212e8024827a4228dd4ca6ed392, State: Initialized, Role: FOLLOWER
I20260807 07:02:14.234858 31819 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } peers { permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 34401 } }
I20260807 07:02:14.235808 31829 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } peers { permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 34401 } } }
I20260807 07:02:14.236135 31829 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 07:02:14.236877 31819 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [sys.catalog]: configured and running, proceeding with master startup.
W20260807 07:02:14.247231 31842 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 07:02:14.247406 31842 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 07:02:14.251788 31721 master_runner.cc:428] Detected that this master a0998212e8024827a4228dd4ca6ed392 is joining an existing cluster
I20260807 07:02:14.251919 31721 master_runner.cc:432] Initiating AddMaster RPC to add 127.21.83.61:34401
I20260807 07:02:14.287418 31824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:14.269688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59008 (local address 127.21.83.62:45341)
0807 07:02:14.269917 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:14.269923 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:14.270824 (+ 901us) server_negotiation.cc:407] Connection header received
0807 07:02:14.271134 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:14.271141 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:14.271248 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:14.271327 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:14.272987 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:14.274196 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:14.276303 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:14.277048 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:14.277161 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:14.279986 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:14.279993 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:14.279998 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:14.280045 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:14.283845 (+ 3800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:14.284652 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:14.284658 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:14.284661 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:14.284826 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:14.285335 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:14.285340 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:14.285345 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:14.285743 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:14.285883 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:14.286510 (+ 627us) server_negotiation.cc:299] Negotiation successful
0807 07:02:14.286796 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260807 07:02:14.288445 31852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:14.269908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:45341 (local address 127.0.0.1:59008)
0807 07:02:14.270690 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:14.270715 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:14.270954 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:14.271565 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:14.271571 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:14.272329 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:14.272752 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:14.272767 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:14.274358 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:14.274364 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:14.276089 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:14.276109 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:14.277493 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:14.277501 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:14.277764 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:14.278484 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:14.278508 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:14.279742 (+ 1234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:14.284030 (+ 4288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:14.284035 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:14.284039 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:14.284513 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:14.285004 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:14.285009 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:14.285013 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:14.285188 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:14.285985 (+ 797us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:14.285990 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:14.286205 (+ 215us) client_negotiation.cc:769] Sending connection context
0807 07:02:14.287661 (+ 1456us) client_negotiation.cc:240] Negotiation successful
0807 07:02:14.288022 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"mutex_wait_us":28,"thread_start_us":409,"threads_started":1}
I20260807 07:02:14.296414 31823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:14.270978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49112 (local address 127.21.83.61:34401)
0807 07:02:14.271163 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:14.271169 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:14.275965 (+ 4796us) server_negotiation.cc:407] Connection header received
0807 07:02:14.276161 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:14.276166 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:14.276257 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:14.276349 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:14.277502 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:14.278701 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:14.280263 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:14.280998 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:14.281098 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:14.286275 (+ 5177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:14.286282 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:14.286286 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:14.286315 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:14.293282 (+ 6967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:14.294196 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:14.294202 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:14.294206 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:14.294387 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:14.294842 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:14.294847 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:14.294850 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:14.295249 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:14.295416 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:14.295876 (+ 460us) server_negotiation.cc:299] Negotiation successful
0807 07:02:14.296113 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":54}
I20260807 07:02:14.298164 31853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:14.270927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.61:34401 (local address 127.0.0.1:49112)
0807 07:02:14.275866 (+ 4939us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:14.275890 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:14.276037 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:14.276364 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:14.276367 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:14.277033 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:14.277349 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:14.277366 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:14.278870 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:14.278874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:14.280084 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:14.280099 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:14.281406 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:14.281412 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:14.281572 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:14.285064 (+ 3492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:14.285087 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:14.286060 (+ 973us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:14.293453 (+ 7393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:14.293458 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:14.293462 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:14.294055 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:14.294523 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:14.294527 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:14.294530 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:14.294697 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:14.295488 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:14.295492 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:14.295636 (+ 144us) client_negotiation.cc:769] Sending connection context
0807 07:02:14.297596 (+ 1960us) client_negotiation.cc:240] Negotiation successful
0807 07:02:14.297847 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4748,"thread_start_us":5175,"threads_started":1}
WARNING: no policy specified for oryx/127.21.83.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.83.60@KRBTEST.COM" created.
Aug 07 07:02:14 dist-test-slave-ncp9 krb5kdc[30984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086131, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.83.62@KRBTEST.COM for oryx/127.21.83.61@KRBTEST.COM
I20260807 07:02:14.331202 31855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:14.296710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.61:34401 (local address 127.0.0.1:49118)
0807 07:02:14.307338 (+ 10628us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:14.307397 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:14.308496 (+ 1099us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:14.309578 (+ 1082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:14.309587 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:14.310501 (+ 914us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:14.310770 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:14.310781 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:14.312584 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:14.312590 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:14.314285 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:14.314304 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:14.315137 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:14.315145 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:14.315414 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:14.316216 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:14.316242 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:14.319001 (+ 2759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:14.327322 (+ 8321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:14.327346 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:14.327357 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:14.328081 (+ 724us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:14.328670 (+ 589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:14.328676 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:14.328680 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:14.328948 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:14.329796 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:14.329807 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:14.330062 (+ 255us) client_negotiation.cc:769] Sending connection context
0807 07:02:14.330294 (+ 232us) client_negotiation.cc:240] Negotiation successful
0807 07:02:14.330702 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10292,"thread_start_us":10895,"threads_started":1}
I20260807 07:02:14.332346 31854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:14.296652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49118 (local address 127.21.83.61:34401)
0807 07:02:14.297291 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:14.297296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:14.307548 (+ 10252us) server_negotiation.cc:407] Connection header received
0807 07:02:14.308823 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:14.308830 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:14.308975 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:14.309070 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:14.311004 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:14.312375 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:14.314454 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:14.314938 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:14.315065 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:14.319262 (+ 4197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:14.319270 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:14.319276 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:14.319314 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:14.327056 (+ 7742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:14.328284 (+ 1228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:14.328295 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:14.328299 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:14.328498 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:14.329103 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:14.329109 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:14.329113 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:14.329545 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:14.329706 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:14.331606 (+ 1900us) server_negotiation.cc:299] Negotiation successful
0807 07:02:14.331886 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":796,"threads_started":1}
I20260807 07:02:14.334578 31048 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.21.83.61:34401: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a0998212e8024827a4228dd4ca6ed392"
member_type: NON_VOTER
last_known_addr {
host: "127.21.83.61"
port: 34401
}
attrs {
promote: true
}
}
dest_uuid: "9c117a2eae7c43dd92d88685ac08f1eb"
cas_config_opid_index: -1
I20260807 07:02:14.336062 31048 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } peers { permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: NON_VOTER last_known_addr { host: "127.21.83.61" port: 34401 } attrs { promote: true } }
I20260807 07:02:14.348078 31784 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:02:14.347915 31859 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER a0998212e8024827a4228dd4ca6ed392 (127.21.83.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } peers { permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: NON_VOTER last_known_addr { host: "127.21.83.61" port: 34401 } attrs { promote: true } } }
I20260807 07:02:14.352257 31784 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [term 1 FOLLOWER]: Refusing update from remote peer 9c117a2eae7c43dd92d88685ac08f1eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260807 07:02:14.353320 31859 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.21.83.61:34401
I20260807 07:02:14.354210 31861 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: NON_VOTER last_known_addr { host: "127.21.83.61" port: 34401 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:02:14.355221 31721 master.cc:537] Master@127.21.83.61:34401 shutting down...
W20260807 07:02:14.358049 31033 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.83.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.60.keytab.
Entry for principal oryx/127.21.83.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.60.keytab.
W20260807 07:02:14.364876 21836 mini_kdc.cc:262] Time spent creating service keytab for oryx/127.21.83.60: real 0.112s user 0.000s sys 0.027s
I20260807 07:02:14.366436 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.60:43239
--webserver_interface=127.21.83.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.60.keytab
--principal=oryx/127.21.83.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:41345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:45341,127.21.83.61:34401,127.21.83.60:43239
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 07:02:14.372381 31860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } peers { permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: NON_VOTER last_known_addr { host: "127.21.83.61" port: 34401 } attrs { promote: true } } }
I20260807 07:02:14.372809 31860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb [sys.catalog]: This master's current role is: LEADER
I20260807 07:02:14.375944 31721 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:02:14.376528 31721 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:02:14.381134 31721 tablet_replica.cc:330] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392: stopping tablet replica
I20260807 07:02:14.389819 31721 master.cc:552] Master@127.21.83.61:34401 shutdown complete.
I20260807 07:02:14.393157 31721 master_runner.cc:305] Clearing existing system tablet
I20260807 07:02:14.393934 31721 fs_manager.cc:359] Metadata directory not provided
I20260807 07:02:14.394114 31721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260807 07:02:14.402387 31033 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.83.61:34401, user_credentials={real_user=slave}} blocked reactor thread for 44357.3us
I20260807 07:02:14.404264 31855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:14.403728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.61:34401 (local address 127.0.0.1:49128)
0807 07:02:14.403919 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:14.404063 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.61:34401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260807 07:02:14.404556 31033 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb -> Peer a0998212e8024827a4228dd4ca6ed392 (127.21.83.61:34401): Couldn't send request to peer a0998212e8024827a4228dd4ca6ed392. Status: Network error: Client connection negotiation failed: client connection to 127.21.83.61:34401: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260807 07:02:14.404672 31721 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.002s sys 0.001s
I20260807 07:02:14.406570 31868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:14.406924 31721 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:02:14.407220 31721 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260807 07:02:14.407311 31721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-1/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-1/wal
uuid: "a0998212e8024827a4228dd4ca6ed392"
format_stamp: "Formatted at 2026-08-07 07:02:14 on dist-test-slave-ncp9"
server_key: "197c164793913b7db3b2b80d1dd99136"
server_key_iv: "34dd029e44952922f49e4d01d11670d2"
server_key_version: "encryptionkey@0"
I20260807 07:02:14.412382 31721 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:02:14.419492 31721 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260807 07:02:14.419697 31721 log.cc:1196] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260807 07:02:14.420166 31721 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392: Deleting consensus metadata
I20260807 07:02:14.420439 31721 master_runner.cc:315] Copying system tablet from 127.21.83.62:45341
I20260807 07:02:14.436585 31721 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392: tablet copy: Beginning tablet copy session from remote peer at address 127.21.83.62:45341
I20260807 07:02:14.454133 31824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:14.437645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59014 (local address 127.21.83.62:45341)
0807 07:02:14.437873 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:14.437880 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:14.438769 (+ 889us) server_negotiation.cc:407] Connection header received
0807 07:02:14.439043 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:14.439049 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:14.439152 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:14.439272 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:14.440552 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:14.441717 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:14.443671 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:14.444327 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:14.444476 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:14.446960 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:14.446967 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:14.446972 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:14.447015 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:14.450772 (+ 3757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:14.451724 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:14.451730 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:14.451733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:14.451914 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:14.452359 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:14.452364 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:14.452367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:14.452805 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:14.452996 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:14.453505 (+ 509us) server_negotiation.cc:299] Negotiation successful
0807 07:02:14.453759 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 07:02:14.454180 31876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:14.438027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:45341 (local address 127.0.0.1:59014)
0807 07:02:14.438636 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:14.438660 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:14.438835 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:14.439498 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:14.439502 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:14.440068 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:14.440374 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:14.440388 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:14.441870 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:14.441876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:14.443463 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:14.443482 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:14.444522 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:14.444529 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:14.444730 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:14.445510 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:14.445536 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:14.446707 (+ 1171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:14.450966 (+ 4259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:14.450973 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:14.450977 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:14.451548 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:14.452046 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:14.452051 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:14.452055 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:14.452249 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:14.452971 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:14.452976 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:14.453197 (+ 221us) client_negotiation.cc:769] Sending connection context
0807 07:02:14.453361 (+ 164us) client_negotiation.cc:240] Negotiation successful
0807 07:02:14.453726 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":269,"threads_started":1}
I20260807 07:02:14.455447 31080 tablet_copy_service.cc:140] P 9c117a2eae7c43dd92d88685ac08f1eb: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a0998212e8024827a4228dd4ca6ed392 ({username='oryx', principal='oryx/127.21.83.61@KRBTEST.COM'} at 127.0.0.1:59014)
I20260807 07:02:14.455771 31080 tablet_copy_service.cc:161] P 9c117a2eae7c43dd92d88685ac08f1eb: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a0998212e8024827a4228dd4ca6ed392 at {username='oryx', principal='oryx/127.21.83.61@KRBTEST.COM'} at 127.0.0.1:59014: session id = a0998212e8024827a4228dd4ca6ed392-00000000000000000000000000000000
I20260807 07:02:14.460598 31080 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 07:02:14.463840 31721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260807 07:02:14.471040 31721 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392: tablet copy: Starting download of 0 data blocks...
I20260807 07:02:14.471408 31721 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392: tablet copy: Starting download of 1 WAL segments...
I20260807 07:02:14.474785 31721 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 07:02:14.479426 31080 tablet_copy_service.cc:342] P 9c117a2eae7c43dd92d88685ac08f1eb: Request end of tablet copy session a0998212e8024827a4228dd4ca6ed392-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.21.83.61@KRBTEST.COM'} at 127.0.0.1:59014
I20260807 07:02:14.479708 31080 tablet_copy_service.cc:433] P 9c117a2eae7c43dd92d88685ac08f1eb: ending tablet copy session a0998212e8024827a4228dd4ca6ed392-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a0998212e8024827a4228dd4ca6ed392
I20260807 07:02:14.492784 31721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:02:14.495808 31877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:14.501698 31878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:02:14.510483 31721 server_base.cc:1034] running on GCE node
W20260807 07:02:14.510890 31880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:02:14.511363 31721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:02:14.512112 31721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:02:14.513480 31721 hybrid_clock.cc:648] HybridClock initialized: now 1786086134513407 us; error 48 us; skew 500 ppm
I20260807 07:02:14.513756 31721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:02:14 dist-test-slave-ncp9 krb5kdc[30984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086134, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.83.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:02:14.516294 31721 init.cc:377] Logged in from keytab as oryx/127.21.83.61@KRBTEST.COM (short username oryx)
I20260807 07:02:14.517855 31721 webserver.cc:460] Webserver started at http://127.21.83.61:44407/ using document root <none> and password file <none>
I20260807 07:02:14.518265 31721 fs_manager.cc:359] Metadata directory not provided
I20260807 07:02:14.518352 31721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:02:14.521694 31721 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260807 07:02:14.523883 31887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:14.524375 31721 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 07:02:14.524525 31721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-1/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-1/wal
uuid: "a0998212e8024827a4228dd4ca6ed392"
format_stamp: "Formatted at 2026-08-07 07:02:14 on dist-test-slave-ncp9"
server_key: "197c164793913b7db3b2b80d1dd99136"
server_key_iv: "34dd029e44952922f49e4d01d11670d2"
server_key_version: "encryptionkey@0"
I20260807 07:02:14.524703 31721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:02:14.541079 31721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:02:14.545403 31721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:02:14.569576 31721 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.61:34401
I20260807 07:02:14.569766 31956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.61:34401 every 8 connection(s)
I20260807 07:02:14.570679 31721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-1/data/info.pb
I20260807 07:02:14.576133 31957 sys_catalog.cc:263] Verifying existing consensus state
I20260807 07:02:14.580251 31957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392: Bootstrap starting.
I20260807 07:02:14.600163 31957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 07:02:14.600802 31957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392: Bootstrap complete.
I20260807 07:02:14.602006 31957 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } peers { permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: NON_VOTER last_known_addr { host: "127.21.83.61" port: 34401 } attrs { promote: true } }
I20260807 07:02:14.602324 31957 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a0998212e8024827a4228dd4ca6ed392, State: Initialized, Role: LEARNER
I20260807 07:02:14.602584 31957 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } peers { permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: NON_VOTER last_known_addr { host: "127.21.83.61" port: 34401 } attrs { promote: true } }
I20260807 07:02:14.603484 31961 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } peers { permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: NON_VOTER last_known_addr { host: "127.21.83.61" port: 34401 } attrs { promote: true } } }
I20260807 07:02:14.603776 31957 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:02:14.603911 31961 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [sys.catalog]: This master's current role is: LEARNER
I20260807 07:02:14.612715 31979 catalog_manager.cc:1259] Loaded cluster ID: b2a37eafb25945a195ac09feead69fd5
I20260807 07:02:14.612816 31979 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392: loading cluster ID for follower catalog manager: success
I20260807 07:02:14.615439 31979 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392: acquiring CA information for follower catalog manager: success
I20260807 07:02:14.617698 31979 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 07:02:14.677667 31711 heartbeater.cc:503] Master 127.21.83.62:45341 was elected leader, sending a full tablet report...
I20260807 07:02:14.864447 31981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:14.854250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49130 (local address 127.21.83.61:34401)
0807 07:02:14.855718 (+ 1468us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:14.855723 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:14.855781 (+ 58us) server_negotiation.cc:407] Connection header received
0807 07:02:14.855849 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:14.855853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:14.855934 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:14.856064 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:02:14.856761 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:14.858776 (+ 2015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:14.861123 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:14.863035 (+ 1912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:14.863201 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:14.863600 (+ 399us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:14.863824 (+ 224us) server_negotiation.cc:299] Negotiation successful
0807 07:02:14.863995 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1288,"thread_start_us":1075,"threads_started":1}
I20260807 07:02:14.864447 31855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:14.853985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.61:34401 (local address 127.0.0.1:49130)
0807 07:02:14.854203 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:14.854228 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:14.854377 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:14.856155 (+ 1778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:14.856162 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:14.856197 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:02:14.856546 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:14.856559 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:14.859082 (+ 2523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:14.859089 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:14.860876 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:14.860933 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:14.863308 (+ 2375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:14.863316 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:14.863604 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:14.863679 (+ 75us) client_negotiation.cc:769] Sending connection context
0807 07:02:14.863783 (+ 104us) client_negotiation.cc:240] Negotiation successful
0807 07:02:14.863971 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260807 07:02:14.865939 31926 raft_consensus.cc:1203] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260807 07:02:14.866776 31961 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c117a2eae7c43dd92d88685ac08f1eb. Latest consensus state: current_term: 1 leader_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } peers { permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: NON_VOTER last_known_addr { host: "127.21.83.61" port: 34401 } attrs { promote: true } } }
I20260807 07:02:14.867009 31961 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [sys.catalog]: This master's current role is: LEARNER
W20260807 07:02:15.029584 31867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:02:15.029980 31867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:02:15.030208 31867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:02:15.041769 31867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:02:15.041980 31867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:02:15.042111 31867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:02:15.042266 31867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:02:15.042383 31867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:02:15.057983 31867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:41345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:45341,127.21.83.61:34401,127.21.83.60:43239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.60.keytab
--principal=oryx/127.21.83.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.60:43239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.83.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:02:15.059762 31867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:02:15.061441 31867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:02:15.080687 31987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:15.081671 31988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:15.081925 31992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:02:15.083946 31867 server_base.cc:1034] running on GCE node
I20260807 07:02:15.084689 31867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:02:15.086198 31867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:02:15.087467 31867 hybrid_clock.cc:648] HybridClock initialized: now 1786086135087324 us; error 58 us; skew 500 ppm
I20260807 07:02:15.087966 31867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:02:15 dist-test-slave-ncp9 krb5kdc[30984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086135, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.83.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:02:15.092598 31867 init.cc:377] Logged in from keytab as oryx/127.21.83.60@KRBTEST.COM (short username oryx)
I20260807 07:02:15.094787 31867 webserver.cc:460] Webserver started at http://127.21.83.60:42967/ using document root <none> and password file <none>
I20260807 07:02:15.095772 31867 fs_manager.cc:359] Metadata directory not provided
I20260807 07:02:15.095948 31867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:02:15.096324 31867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:02:15.100961 31867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-2/data/instance:
uuid: "3ce46c8da37c42dbb13691c7fe185e55"
format_stamp: "Formatted at 2026-08-07 07:02:15 on dist-test-slave-ncp9"
server_key: "e941d38c07cc0c718d7119f263689faf"
server_key_iv: "2d3b5b0ed8a430173b0c0fae66eeb8fb"
server_key_version: "encryptionkey@0"
I20260807 07:02:15.102262 31867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-2/wal/instance:
uuid: "3ce46c8da37c42dbb13691c7fe185e55"
format_stamp: "Formatted at 2026-08-07 07:02:15 on dist-test-slave-ncp9"
server_key: "e941d38c07cc0c718d7119f263689faf"
server_key_iv: "2d3b5b0ed8a430173b0c0fae66eeb8fb"
server_key_version: "encryptionkey@0"
I20260807 07:02:15.109872 31867 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.002s
I20260807 07:02:15.114894 32003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:15.116328 31867 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260807 07:02:15.116568 31867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-2/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-2/wal
uuid: "3ce46c8da37c42dbb13691c7fe185e55"
format_stamp: "Formatted at 2026-08-07 07:02:15 on dist-test-slave-ncp9"
server_key: "e941d38c07cc0c718d7119f263689faf"
server_key_iv: "2d3b5b0ed8a430173b0c0fae66eeb8fb"
server_key_version: "encryptionkey@0"
I20260807 07:02:15.116879 31867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:02:15.136238 31867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:02:15.142050 31867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:02:15.142529 31867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:02:15.194072 31867 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.60:43239
I20260807 07:02:15.194375 32072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.60:43239 every 8 connection(s)
I20260807 07:02:15.195673 31867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-2/data/info.pb
I20260807 07:02:15.201189 32073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:02:15.203589 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 31867
I20260807 07:02:15.216051 32073 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 07:02:15 dist-test-slave-ncp9 krb5kdc[30984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.83.60@KRBTEST.COM
Aug 07 07:02:15 dist-test-slave-ncp9 krb5kdc[30984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086135, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.83.60@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM
I20260807 07:02:15.240048 32076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:15.206576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48512 (local address 127.21.83.60:43239)
0807 07:02:15.208704 (+ 2128us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:15.208721 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:15.208797 (+ 76us) server_negotiation.cc:407] Connection header received
0807 07:02:15.211839 (+ 3042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:15.211876 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:15.212558 (+ 682us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:15.213197 (+ 639us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:15.215629 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:15.218575 (+ 2946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:15.221140 (+ 2565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:15.222703 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:15.222945 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:15.229497 (+ 6552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:15.229506 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:15.229520 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:15.229559 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:15.234614 (+ 5055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:15.235619 (+ 1005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:15.235636 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:15.235642 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:15.235863 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:15.236445 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:15.236448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:15.236450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:15.237288 (+ 838us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:15.237641 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:15.238082 (+ 441us) server_negotiation.cc:299] Negotiation successful
0807 07:02:15.238501 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1795,"thread_start_us":1321,"threads_started":1}
I20260807 07:02:15.245122 32080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:15.220772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:45341 (local address 127.0.0.1:59022)
0807 07:02:15.223025 (+ 2253us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:15.223080 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:15.223301 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:15.223684 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:15.223691 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:15.224789 (+ 1098us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:15.225666 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:15.225679 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:15.227106 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:15.227109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:15.229095 (+ 1986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:15.229109 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:15.230226 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:15.230237 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:15.230497 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:15.231318 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:15.231339 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:15.235353 (+ 4014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:15.241157 (+ 5804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:15.241167 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:15.241173 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:15.241947 (+ 774us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:15.242515 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:15.242518 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:15.242521 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:15.242773 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:15.243435 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:15.243442 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:15.243773 (+ 331us) client_negotiation.cc:769] Sending connection context
0807 07:02:15.243973 (+ 200us) client_negotiation.cc:240] Negotiation successful
0807 07:02:15.244377 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1857,"thread_start_us":1417,"threads_started":1}
I20260807 07:02:15.245110 32077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:15.220305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59022 (local address 127.21.83.62:45341)
0807 07:02:15.221437 (+ 1132us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:15.221444 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:15.223182 (+ 1738us) server_negotiation.cc:407] Connection header received
0807 07:02:15.223427 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:15.223434 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:15.223503 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:15.223593 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:15.225811 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:15.226930 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:15.229252 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:15.229854 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:15.229971 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:15.235593 (+ 5622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:15.235600 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:15.235605 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:15.235643 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:15.241014 (+ 5371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:15.242089 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:15.242095 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:15.242099 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:15.242391 (+ 292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:15.242900 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:15.242906 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:15.242910 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:15.243299 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:15.243443 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:15.244117 (+ 674us) server_negotiation.cc:299] Negotiation successful
0807 07:02:15.244436 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":977,"thread_start_us":785,"threads_started":1}
I20260807 07:02:15.246778 32073 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.83.61" port: 34401 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 07:02:15 dist-test-slave-ncp9 krb5kdc[30984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086135, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.83.60@KRBTEST.COM for oryx/127.21.83.61@KRBTEST.COM
I20260807 07:02:15.266090 32080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:15.247482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.61:34401 (local address 127.0.0.1:49142)
0807 07:02:15.247755 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:15.247776 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:15.247885 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:15.248269 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:15.248272 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:15.248862 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:15.249182 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:15.249193 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:15.250538 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:15.250542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:15.252185 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:15.252200 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:15.253555 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:15.253562 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:15.253831 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:15.254733 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:15.254755 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:15.258002 (+ 3247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:15.263064 (+ 5062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:15.263069 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:15.263072 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:15.263759 (+ 687us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:15.264137 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:15.264140 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:15.264142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:15.264437 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:15.265124 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:15.265127 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:15.265281 (+ 154us) client_negotiation.cc:769] Sending connection context
0807 07:02:15.265395 (+ 114us) client_negotiation.cc:240] Negotiation successful
0807 07:02:15.265713 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260807 07:02:15.266260 31981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:15.247834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49142 (local address 127.21.83.61:34401)
0807 07:02:15.248021 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:15.248026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:15.248045 (+ 19us) server_negotiation.cc:407] Connection header received
0807 07:02:15.248097 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:15.248100 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:15.248163 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:15.248242 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:15.249353 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:15.250352 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:15.252373 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:15.253302 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:15.253411 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:15.258319 (+ 4908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:15.258327 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:15.258330 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:15.258372 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:15.262888 (+ 4516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:15.263865 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:15.263868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:15.263871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:15.264045 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:15.264524 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:15.264527 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:15.264529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:15.265014 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:15.265132 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:15.265508 (+ 376us) server_negotiation.cc:299] Negotiation successful
0807 07:02:15.265766 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 07:02:15.267390 32073 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.83.60" port: 43239 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 07:02:15 dist-test-slave-ncp9 krb5kdc[30984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086135, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.83.60@KRBTEST.COM for oryx/127.21.83.60@KRBTEST.COM
I20260807 07:02:15.285898 32080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:15.269117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:43239 (local address 127.0.0.1:48520)
0807 07:02:15.269501 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:15.269526 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:15.269655 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:15.269965 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:15.269970 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:15.270788 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:15.271048 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:15.271059 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:15.272367 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:15.272372 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:15.273767 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:15.273780 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:15.274888 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:15.274893 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:15.275239 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:15.276012 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:15.276030 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:15.278857 (+ 2827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:15.283076 (+ 4219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:15.283079 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:15.283082 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:15.283537 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:15.283929 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:15.283932 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:15.283934 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:15.284221 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:15.284853 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:15.284856 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:15.285118 (+ 262us) client_negotiation.cc:769] Sending connection context
0807 07:02:15.285268 (+ 150us) client_negotiation.cc:240] Negotiation successful
0807 07:02:15.285455 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260807 07:02:15.285954 32076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:15.269009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48520 (local address 127.21.83.60:43239)
0807 07:02:15.269217 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:15.269223 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:15.269602 (+ 379us) server_negotiation.cc:407] Connection header received
0807 07:02:15.269778 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:15.269782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:15.269840 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:15.269916 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:15.271210 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:15.272237 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:15.273909 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:15.274619 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:15.274787 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:15.279058 (+ 4271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:15.279062 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:15.279065 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:15.279095 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:15.282917 (+ 3822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:15.283639 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:15.283643 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:15.283645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:15.283838 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:15.284319 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:15.284322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:15.284324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:15.284755 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:15.284865 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:15.285416 (+ 551us) server_negotiation.cc:299] Negotiation successful
0807 07:02:15.285596 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 07:02:15.292840 32073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55: Bootstrap starting.
I20260807 07:02:15.297686 32073 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55: Neither blocks nor log segments found. Creating new log.
I20260807 07:02:15.298910 32073 log.cc:824] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55: Log is configured to *not* fsync() on all Append() calls
I20260807 07:02:15.302757 32073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55: No bootstrap required, opened a new log
I20260807 07:02:15.316187 32073 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } peers { permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 34401 } } peers { permanent_uuid: "3ce46c8da37c42dbb13691c7fe185e55" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 43239 } }
I20260807 07:02:15.316747 32073 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:02:15.317005 32073 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ce46c8da37c42dbb13691c7fe185e55, State: Initialized, Role: FOLLOWER
I20260807 07:02:15.317667 32073 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } peers { permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 34401 } } peers { permanent_uuid: "3ce46c8da37c42dbb13691c7fe185e55" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 43239 } }
I20260807 07:02:15.319355 32082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } peers { permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 34401 } } peers { permanent_uuid: "3ce46c8da37c42dbb13691c7fe185e55" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 43239 } } }
I20260807 07:02:15.320283 32082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 07:02:15.320531 32073 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55 [sys.catalog]: configured and running, proceeding with master startup.
W20260807 07:02:15.338613 32096 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 07:02:15.338829 32096 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 07:02:15.342362 31867 master_runner.cc:428] Detected that this master 3ce46c8da37c42dbb13691c7fe185e55 is joining an existing cluster
I20260807 07:02:15.342605 31867 master_runner.cc:432] Initiating AddMaster RPC to add 127.21.83.60:43239
I20260807 07:02:15.347263 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 31140
I20260807 07:02:15.364413 32082 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:02:15.365072 32082 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } peers { permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 34401 } } peers { permanent_uuid: "3ce46c8da37c42dbb13691c7fe185e55" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 43239 } }
I20260807 07:02:15.368592 32082 leader_election.cc:288] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c117a2eae7c43dd92d88685ac08f1eb (127.21.83.62:45341), a0998212e8024827a4228dd4ca6ed392 (127.21.83.61:34401)
I20260807 07:02:15.369594 31063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3ce46c8da37c42dbb13691c7fe185e55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" is_pre_election: true
I20260807 07:02:15.370904 31926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3ce46c8da37c42dbb13691c7fe185e55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0998212e8024827a4228dd4ca6ed392" is_pre_election: true
I20260807 07:02:15.372443 32008 leader_election.cc:302] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ce46c8da37c42dbb13691c7fe185e55; no voters: 9c117a2eae7c43dd92d88685ac08f1eb, a0998212e8024827a4228dd4ca6ed392
I20260807 07:02:15.373807 32082 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
Aug 07 07:02:15 dist-test-slave-ncp9 krb5kdc[30984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.83.1@KRBTEST.COM
I20260807 07:02:15.391002 32104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:15.348974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36836 (local address 127.21.83.1:39755)
0807 07:02:15.350228 (+ 1254us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:15.350235 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:15.350259 (+ 24us) server_negotiation.cc:407] Connection header received
0807 07:02:15.350342 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:15.350352 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:15.350652 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:15.353890 (+ 3238us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:15.354409 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:15.356711 (+ 2302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:15.360499 (+ 3788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:15.362257 (+ 1758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:15.378973 (+ 16716us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:15.379517 (+ 544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:15.379532 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:15.379553 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:15.379596 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:15.385171 (+ 5575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:15.386547 (+ 1376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:15.386558 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:15.386562 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:15.386853 (+ 291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:15.387485 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:15.387490 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:15.387503 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:15.388050 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:15.389998 (+ 1948us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:15.390206 (+ 208us) server_negotiation.cc:299] Negotiation successful
0807 07:02:15.390503 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1107,"thread_start_us":832,"threads_started":1}
I20260807 07:02:15.393407 31319 generic_service.cc:196] Checking for leaks (request via RPC)
W20260807 07:02:15.396257 32010 tablet.cc:2369] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 07:02:15.403806 32076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:15.351390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48534 (local address 127.21.83.60:43239)
0807 07:02:15.351543 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:15.351550 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:15.372153 (+ 20603us) server_negotiation.cc:407] Connection header received
0807 07:02:15.372588 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:15.372593 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:15.372813 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:15.372926 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:15.375764 (+ 2838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:15.377238 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:15.383402 (+ 6164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:15.385087 (+ 1685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:15.385234 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:15.393564 (+ 8330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:15.393574 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:15.393579 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:15.393625 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:15.399351 (+ 5726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:15.400561 (+ 1210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:15.400567 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:15.400570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:15.400840 (+ 270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:15.401394 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:15.401401 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:15.401404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:15.402047 (+ 643us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:15.402207 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:15.402986 (+ 779us) server_negotiation.cc:299] Negotiation successful
0807 07:02:15.403279 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260807 07:02:15.406575 32105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:15.352038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:43239 (local address 127.0.0.1:48534)
0807 07:02:15.372050 (+ 20012us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:15.372075 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:15.372417 (+ 342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:15.373262 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:15.373266 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:15.375060 (+ 1794us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:15.375610 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:15.375625 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:15.377664 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:15.377669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:15.383202 (+ 5533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:15.383222 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:15.387322 (+ 4100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:15.387332 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:15.387947 (+ 615us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:15.388950 (+ 1003us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:15.388974 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:15.393293 (+ 4319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:15.399567 (+ 6274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:15.399572 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:15.399577 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:15.400422 (+ 845us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:15.401037 (+ 615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:15.401042 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:15.401046 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:15.401275 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:15.402297 (+ 1022us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:15.402302 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:15.402600 (+ 298us) client_negotiation.cc:769] Sending connection context
0807 07:02:15.404132 (+ 1532us) client_negotiation.cc:240] Negotiation successful
0807 07:02:15.404609 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19792,"mutex_wait_us":1,"thread_start_us":17270,"threads_started":1}
I20260807 07:02:15.412043 32077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:15.351329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59028 (local address 127.21.83.62:45341)
0807 07:02:15.353690 (+ 2361us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:15.353697 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:15.371543 (+ 17846us) server_negotiation.cc:407] Connection header received
0807 07:02:15.371619 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:15.371624 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:15.371718 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:15.371831 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:15.375444 (+ 3613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:15.376987 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:15.384948 (+ 7961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:15.386044 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:15.386156 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:15.390471 (+ 4315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:15.390477 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:15.390481 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:15.390520 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:15.395663 (+ 5143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:15.406430 (+ 10767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:15.406437 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:15.406441 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:15.407728 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:15.408523 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:15.408531 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:15.408535 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:15.409172 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:15.409349 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:15.410918 (+ 1569us) server_negotiation.cc:299] Negotiation successful
0807 07:02:15.411319 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2182}
I20260807 07:02:15.414330 32106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:15.351299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:45341 (local address 127.0.0.1:59028)
0807 07:02:15.369884 (+ 18585us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:15.369905 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:15.370087 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:15.371873 (+ 1786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:15.371878 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:15.374448 (+ 2570us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:15.375200 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:15.375217 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:15.377247 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:15.377252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:15.384707 (+ 7455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:15.384724 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:15.386542 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:15.386552 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:15.386877 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:15.388000 (+ 1123us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:15.388106 (+ 106us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:15.389996 (+ 1890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:15.395851 (+ 5855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:15.395856 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:15.395860 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:15.406223 (+ 10363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:15.407920 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:15.407926 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:15.407931 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:15.408344 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:15.409464 (+ 1120us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:15.409469 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:15.409914 (+ 445us) client_negotiation.cc:769] Sending connection context
0807 07:02:15.412297 (+ 2383us) client_negotiation.cc:240] Negotiation successful
0807 07:02:15.412953 (+ 656us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18341,"mutex_wait_us":256,"spinlock_wait_cycles":35712,"thread_start_us":19419,"threads_started":1}
I20260807 07:02:15.419154 32108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:15.372089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.61:34401 (local address 127.0.0.1:49154)
0807 07:02:15.383506 (+ 11417us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:15.383529 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:15.383776 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:15.384469 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:15.384475 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:15.386632 (+ 2157us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:15.391275 (+ 4643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:15.391292 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:15.393088 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:15.393094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:15.399861 (+ 6767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:15.399877 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:15.402185 (+ 2308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:15.402211 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:15.402595 (+ 384us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:15.403637 (+ 1042us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:15.403659 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:15.405578 (+ 1919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:15.413399 (+ 7821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:15.413405 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:15.413409 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:15.414125 (+ 716us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:15.416657 (+ 2532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:15.416664 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:15.416668 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:15.416933 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:15.417672 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:15.417677 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:15.417949 (+ 272us) client_negotiation.cc:769] Sending connection context
0807 07:02:15.418096 (+ 147us) client_negotiation.cc:240] Negotiation successful
0807 07:02:15.418504 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10186,"thread_start_us":7518,"threads_started":1}
I20260807 07:02:15.424211 32111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } peers { permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: NON_VOTER last_known_addr { host: "127.21.83.61" port: 34401 } attrs { promote: true } } }
I20260807 07:02:15.424801 32111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb [sys.catalog]: This master's current role is: LEADER
I20260807 07:02:15.427462 31981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:15.371728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49154 (local address 127.21.83.61:34401)
0807 07:02:15.371922 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:15.371927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:15.383628 (+ 11701us) server_negotiation.cc:407] Connection header received
0807 07:02:15.383953 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:15.383960 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:15.384064 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:15.384143 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:15.391508 (+ 7365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:15.392795 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:15.400093 (+ 7298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:15.401388 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:15.401516 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:15.405832 (+ 4316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:15.405840 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:15.405847 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:15.405915 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:15.413212 (+ 7297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:15.414298 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:15.414305 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:15.414309 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:15.416484 (+ 2175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:15.417117 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:15.417123 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:15.417128 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:15.417528 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:15.425251 (+ 7723us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:15.425477 (+ 226us) server_negotiation.cc:299] Negotiation successful
0807 07:02:15.426295 (+ 818us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 07:02:15.443522 32112 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb: attempting to promote NON_VOTER a0998212e8024827a4228dd4ca6ed392 to VOTER
I20260807 07:02:15.445354 32112 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } peers { permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 34401 } attrs { promote: false } }
Aug 07 07:02:15 dist-test-slave-ncp9 krb5kdc[30984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086131, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.83.62@KRBTEST.COM for oryx/127.21.83.60@KRBTEST.COM
I20260807 07:02:15.461843 31926 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [term 1 LEARNER]: Refusing update from remote peer 9c117a2eae7c43dd92d88685ac08f1eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260807 07:02:15.463037 32114 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 34401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 07:02:15.473798 32076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:15.423644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48540 (local address 127.21.83.60:43239)
0807 07:02:15.423941 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:15.423947 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:15.433655 (+ 9708us) server_negotiation.cc:407] Connection header received
0807 07:02:15.435502 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:15.435508 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:15.435608 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:15.435685 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:15.439119 (+ 3434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:15.440579 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:15.448261 (+ 7682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:15.450622 (+ 2361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:15.450751 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:15.458534 (+ 7783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:15.458544 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:15.458549 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:15.458596 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:15.466880 (+ 8284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:15.469732 (+ 2852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:15.469740 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:15.469743 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:15.470006 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:15.471326 (+ 1320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:15.471332 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:15.471336 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:15.471976 (+ 640us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:15.472148 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:15.472810 (+ 662us) server_negotiation.cc:299] Negotiation successful
0807 07:02:15.473195 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260807 07:02:15.475382 32116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:15.423237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:43239 (local address 127.0.0.1:48540)
0807 07:02:15.433530 (+ 10293us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:15.433553 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:15.435260 (+ 1707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:15.435974 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:15.435980 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:15.437212 (+ 1232us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:15.437929 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:15.437946 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:15.440795 (+ 2849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:15.440802 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:15.448021 (+ 7219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:15.448044 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:15.451539 (+ 3495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:15.451547 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:15.451779 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:15.453553 (+ 1774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:15.453578 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:15.458207 (+ 4629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:15.468454 (+ 10247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:15.468461 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:15.468466 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:15.469532 (+ 1066us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:15.470187 (+ 655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:15.470193 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:15.470196 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:15.471094 (+ 898us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:15.472245 (+ 1151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:15.472250 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:15.472482 (+ 232us) client_negotiation.cc:769] Sending connection context
0807 07:02:15.474308 (+ 1826us) client_negotiation.cc:240] Negotiation successful
0807 07:02:15.474726 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10131,"thread_start_us":10925,"threads_started":1}
I20260807 07:02:15.477638 31048 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.21.83.60:43239: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3ce46c8da37c42dbb13691c7fe185e55"
member_type: NON_VOTER
last_known_addr {
host: "127.21.83.60"
port: 43239
}
attrs {
promote: true
}
}
dest_uuid: "9c117a2eae7c43dd92d88685ac08f1eb"
cas_config_opid_index: 5
W20260807 07:02:15.478546 31048 master_service.cc:301] AddMaster 127.21.83.60:43239 failed with retriable error: Illegal state: Failed initiating master Raft ChangeConfig request, error: unknown: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } peers { permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: NON_VOTER last_known_addr { host: "127.21.83.61" port: 34401 } attrs { promote: true } }.
Pending config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } peers { permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 34401 } attrs { promote: false } }
I20260807 07:02:15.479031 32112 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a0998212e8024827a4228dd4ca6ed392 (127.21.83.61) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } peers { permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 34401 } attrs { promote: false } } }
W20260807 07:02:15.480690 32101 master_proxy_rpc.cc:203] Re-attempting AddMaster request to leader Master (127.21.83.62:45341)
I20260807 07:02:15.484814 32112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } peers { permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 34401 } attrs { promote: false } } }
I20260807 07:02:15.484846 32111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } peers { permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 34401 } attrs { promote: false } } }
I20260807 07:02:15.485208 32112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb [sys.catalog]: This master's current role is: LEADER
I20260807 07:02:15.485319 32111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb [sys.catalog]: This master's current role is: LEADER
I20260807 07:02:15.487290 31926 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a0998212e8024827a4228dd4ca6ed392 (127.21.83.61) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } peers { permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 34401 } attrs { promote: false } } }
I20260807 07:02:15.489215 31048 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.21.83.60:43239: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3ce46c8da37c42dbb13691c7fe185e55"
member_type: NON_VOTER
last_known_addr {
host: "127.21.83.60"
port: 43239
}
attrs {
promote: true
}
}
dest_uuid: "9c117a2eae7c43dd92d88685ac08f1eb"
cas_config_opid_index: 6
I20260807 07:02:15.490080 31048 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } peers { permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "3ce46c8da37c42dbb13691c7fe185e55" member_type: NON_VOTER last_known_addr { host: "127.21.83.60" port: 43239 } attrs { promote: true } }
I20260807 07:02:15.493745 32040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:02:15.495708 31925 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [term 1 FOLLOWER]: Refusing update from remote peer 9c117a2eae7c43dd92d88685ac08f1eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260807 07:02:15.496737 32111 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 34401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260807 07:02:15.496703 32123 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } peers { permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 34401 } attrs { promote: false } } }
I20260807 07:02:15.497548 32123 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 07:02:15.498742 32040 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55 [term 1 FOLLOWER]: Refusing update from remote peer 9c117a2eae7c43dd92d88685ac08f1eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260807 07:02:15.499224 32112 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3ce46c8da37c42dbb13691c7fe185e55 (127.21.83.60) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } peers { permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "3ce46c8da37c42dbb13691c7fe185e55" member_type: NON_VOTER last_known_addr { host: "127.21.83.60" port: 43239 } attrs { promote: true } } }
I20260807 07:02:15.503103 31925 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3ce46c8da37c42dbb13691c7fe185e55 (127.21.83.60) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } peers { permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "3ce46c8da37c42dbb13691c7fe185e55" member_type: NON_VOTER last_known_addr { host: "127.21.83.60" port: 43239 } attrs { promote: true } } }
I20260807 07:02:15.507830 32111 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ce46c8da37c42dbb13691c7fe185e55" member_type: NON_VOTER last_known_addr { host: "127.21.83.60" port: 43239 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:02:15.508846 32112 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.21.83.60:43239
I20260807 07:02:15.511437 31867 master.cc:537] Master@127.21.83.60:43239 shutting down...
I20260807 07:02:15.512125 32114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } peers { permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "3ce46c8da37c42dbb13691c7fe185e55" member_type: NON_VOTER last_known_addr { host: "127.21.83.60" port: 43239 } attrs { promote: true } } }
I20260807 07:02:15.512701 32114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb [sys.catalog]: This master's current role is: LEADER
I20260807 07:02:15.513053 32114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } peers { permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "3ce46c8da37c42dbb13691c7fe185e55" member_type: NON_VOTER last_known_addr { host: "127.21.83.60" port: 43239 } attrs { promote: true } } }
I20260807 07:02:15.513279 32114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb [sys.catalog]: This master's current role is: LEADER
I20260807 07:02:15.513620 32123 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } peers { permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "3ce46c8da37c42dbb13691c7fe185e55" member_type: NON_VOTER last_known_addr { host: "127.21.83.60" port: 43239 } attrs { promote: true } } }
I20260807 07:02:15.513934 32123 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [sys.catalog]: This master's current role is: FOLLOWER
W20260807 07:02:15.519162 31030 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb -> Peer 3ce46c8da37c42dbb13691c7fe185e55 (127.21.83.60:43239): Couldn't send request to peer 3ce46c8da37c42dbb13691c7fe185e55. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260807 07:02:15.523428 31867 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:02:15.523864 31867 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:02:15.524080 31867 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55: stopping tablet replica
I20260807 07:02:15.530835 31867 master.cc:552] Master@127.21.83.60:43239 shutdown complete.
I20260807 07:02:15.530938 31867 master_runner.cc:305] Clearing existing system tablet
I20260807 07:02:15.531292 31867 fs_manager.cc:359] Metadata directory not provided
I20260807 07:02:15.531404 31867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:02:15.533964 31867 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260807 07:02:15.535634 32128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:15.536026 31867 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:02:15.536309 31867 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260807 07:02:15.536417 31867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-2/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-2/wal
uuid: "3ce46c8da37c42dbb13691c7fe185e55"
format_stamp: "Formatted at 2026-08-07 07:02:15 on dist-test-slave-ncp9"
server_key: "e941d38c07cc0c718d7119f263689faf"
server_key_iv: "2d3b5b0ed8a430173b0c0fae66eeb8fb"
server_key_version: "encryptionkey@0"
I20260807 07:02:15.540172 31867 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:02:15.546080 31867 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260807 07:02:15.546420 31867 log.cc:1196] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260807 07:02:15.546880 31867 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55: Deleting consensus metadata
I20260807 07:02:15.547119 31867 master_runner.cc:315] Copying system tablet from 127.21.83.62:45341
I20260807 07:02:15.553436 31867 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55: tablet copy: Beginning tablet copy session from remote peer at address 127.21.83.62:45341
I20260807 07:02:15.569792 32138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:15.554245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:45341 (local address 127.0.0.1:59032)
0807 07:02:15.555247 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:15.555264 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:15.555426 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:15.555794 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:15.555799 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:15.556405 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:15.557060 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:15.557075 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:15.558187 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:15.558191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:15.559425 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:15.559439 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:15.560358 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:15.560364 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:15.560606 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:15.561346 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:15.561365 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:15.562576 (+ 1211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:15.566542 (+ 3966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:15.566549 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:15.566554 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:15.567121 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:15.567554 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:15.567564 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:15.567568 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:15.567841 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:15.568388 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:15.568394 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:15.568640 (+ 246us) client_negotiation.cc:769] Sending connection context
0807 07:02:15.568796 (+ 156us) client_negotiation.cc:240] Negotiation successful
0807 07:02:15.569261 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"thread_start_us":680,"threads_started":1}
I20260807 07:02:15.571251 32077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:15.554291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59032 (local address 127.21.83.62:45341)
0807 07:02:15.554555 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:15.554563 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:15.555354 (+ 791us) server_negotiation.cc:407] Connection header received
0807 07:02:15.555579 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:15.555585 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:15.555657 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:15.555741 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:15.557217 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:15.557980 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:15.559582 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:15.560172 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:15.560342 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:15.562799 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:15.562805 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:15.562808 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:15.562844 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:15.566340 (+ 3496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:15.567258 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:15.567264 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:15.567267 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:15.567432 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:15.567960 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:15.567964 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:15.567966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:15.568263 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:15.568449 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:15.570189 (+ 1740us) server_negotiation.cc:299] Negotiation successful
0807 07:02:15.570462 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260807 07:02:15.575016 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 31140
I20260807 07:02:15.589684 31080 tablet_copy_service.cc:140] P 9c117a2eae7c43dd92d88685ac08f1eb: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3ce46c8da37c42dbb13691c7fe185e55 ({username='oryx', principal='oryx/127.21.83.60@KRBTEST.COM'} at 127.0.0.1:59032)
I20260807 07:02:15.589916 31080 tablet_copy_service.cc:161] P 9c117a2eae7c43dd92d88685ac08f1eb: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3ce46c8da37c42dbb13691c7fe185e55 at {username='oryx', principal='oryx/127.21.83.60@KRBTEST.COM'} at 127.0.0.1:59032: session id = 3ce46c8da37c42dbb13691c7fe185e55-00000000000000000000000000000000
I20260807 07:02:15.593672 31080 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 07:02:15.595934 31867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260807 07:02:15.603550 31867 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55: tablet copy: Starting download of 0 data blocks...
I20260807 07:02:15.604218 31867 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55: tablet copy: Starting download of 1 WAL segments...
I20260807 07:02:15.607450 31867 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 07:02:15.608489 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.1:39755
--local_ip_for_outbound_sockets=127.21.83.1
--tserver_master_addrs=127.21.83.62:45341
--webserver_port=45839
--webserver_interface=127.21.83.1
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.1.keytab
--principal=oryx/127.21.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:41345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.83.62:45341,127.21.83.61:34401
--tserver_master_addrs=127.21.83.62:45341,127.21.83.61:34401,127.21.83.60:43239 with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 07:02:15.611310 31080 tablet_copy_service.cc:342] P 9c117a2eae7c43dd92d88685ac08f1eb: Request end of tablet copy session 3ce46c8da37c42dbb13691c7fe185e55-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.21.83.60@KRBTEST.COM'} at 127.0.0.1:59032
I20260807 07:02:15.611450 31080 tablet_copy_service.cc:433] P 9c117a2eae7c43dd92d88685ac08f1eb: ending tablet copy session 3ce46c8da37c42dbb13691c7fe185e55-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3ce46c8da37c42dbb13691c7fe185e55
I20260807 07:02:15.626590 31867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:02:15.629379 32140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:15.635250 32144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:15.642522 32146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:02:15.646565 31867 server_base.cc:1034] running on GCE node
I20260807 07:02:15.647006 31867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:02:15.647753 31867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:02:15.651397 31867 hybrid_clock.cc:648] HybridClock initialized: now 1786086135651329 us; error 44 us; skew 500 ppm
I20260807 07:02:15.651681 31867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:02:15 dist-test-slave-ncp9 krb5kdc[30984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086135, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.83.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:02:15.653844 31867 init.cc:377] Logged in from keytab as oryx/127.21.83.60@KRBTEST.COM (short username oryx)
I20260807 07:02:15.660483 31867 webserver.cc:460] Webserver started at http://127.21.83.60:33411/ using document root <none> and password file <none>
I20260807 07:02:15.660905 31867 fs_manager.cc:359] Metadata directory not provided
I20260807 07:02:15.661041 31867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:02:15.664490 31867 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260807 07:02:15.666775 32154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:15.667285 31867 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 07:02:15.667425 31867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-2/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-2/wal
uuid: "3ce46c8da37c42dbb13691c7fe185e55"
format_stamp: "Formatted at 2026-08-07 07:02:15 on dist-test-slave-ncp9"
server_key: "e941d38c07cc0c718d7119f263689faf"
server_key_iv: "2d3b5b0ed8a430173b0c0fae66eeb8fb"
server_key_version: "encryptionkey@0"
I20260807 07:02:15.667630 31867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:02:15.695257 31867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:02:15.699514 31867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:02:15.731017 31867 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.60:43239
I20260807 07:02:15.731078 32217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.60:43239 every 8 connection(s)
I20260807 07:02:15.732321 31867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/master-2/data/info.pb
I20260807 07:02:15.738229 32218 sys_catalog.cc:263] Verifying existing consensus state
I20260807 07:02:15.742468 32218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55: Bootstrap starting.
I20260807 07:02:15.762812 32218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 07:02:15.763477 32218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55: Bootstrap complete.
I20260807 07:02:15.764688 32218 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } peers { permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "3ce46c8da37c42dbb13691c7fe185e55" member_type: NON_VOTER last_known_addr { host: "127.21.83.60" port: 43239 } attrs { promote: true } }
I20260807 07:02:15.765049 32218 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3ce46c8da37c42dbb13691c7fe185e55, State: Initialized, Role: LEARNER
I20260807 07:02:15.765352 32218 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } peers { permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "3ce46c8da37c42dbb13691c7fe185e55" member_type: NON_VOTER last_known_addr { host: "127.21.83.60" port: 43239 } attrs { promote: true } }
I20260807 07:02:15.766595 32218 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:02:15.766309 32224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } peers { permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "3ce46c8da37c42dbb13691c7fe185e55" member_type: NON_VOTER last_known_addr { host: "127.21.83.60" port: 43239 } attrs { promote: true } } }
I20260807 07:02:15.766887 32224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55 [sys.catalog]: This master's current role is: LEARNER
I20260807 07:02:15.776343 32237 catalog_manager.cc:1259] Loaded cluster ID: b2a37eafb25945a195ac09feead69fd5
I20260807 07:02:15.776468 32237 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55: loading cluster ID for follower catalog manager: success
I20260807 07:02:15.780108 32237 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55: acquiring CA information for follower catalog manager: success
I20260807 07:02:15.782963 32237 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 07:02:16.092484 32242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:16.085803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:43239 (local address 127.0.0.1:48544)
0807 07:02:16.086530 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:16.086546 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:16.086693 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:16.087045 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:16.087051 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:16.087079 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:02:16.087284 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:16.087298 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:16.088957 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:16.088963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:16.090124 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:16.090138 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:16.091645 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:16.091652 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:16.091845 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:16.091880 (+ 35us) client_negotiation.cc:769] Sending connection context
0807 07:02:16.091976 (+ 96us) client_negotiation.cc:240] Negotiation successful
0807 07:02:16.092132 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":532,"threads_started":1}
I20260807 07:02:16.092854 32243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:16.085896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48544 (local address 127.21.83.60:43239)
0807 07:02:16.086406 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:16.086412 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:16.086631 (+ 219us) server_negotiation.cc:407] Connection header received
0807 07:02:16.086825 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:16.086830 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:16.086907 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:16.087012 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:02:16.087402 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:16.088787 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:16.090290 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:16.091415 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:16.091579 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:16.091861 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:16.092014 (+ 153us) server_negotiation.cc:299] Negotiation successful
0807 07:02:16.092201 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":245,"threads_started":1}
I20260807 07:02:16.094587 32189 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.7] Dedup: 1.7->[]
I20260807 07:02:16.095431 32224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c117a2eae7c43dd92d88685ac08f1eb. Latest consensus state: current_term: 1 leader_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } peers { permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "3ce46c8da37c42dbb13691c7fe185e55" member_type: NON_VOTER last_known_addr { host: "127.21.83.60" port: 43239 } attrs { promote: true } } }
I20260807 07:02:16.095767 32224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55 [sys.catalog]: This master's current role is: LEARNER
W20260807 07:02:16.097496 32139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:02:16.097910 32139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:02:16.098100 32139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:02:16.110394 32139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:02:16.110553 32139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:02:16.110816 32139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.1
I20260807 07:02:16.125409 32139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:41345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.1.keytab
--principal=oryx/127.21.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.1:39755
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.83.1
--webserver_port=45839
--tserver_master_addrs=127.21.83.62:45341,127.21.83.61:34401,127.21.83.60:43239
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.1
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:02:16.126976 32139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:02:16.128589 32139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:02:16.147382 32246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:16.147799 32249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:16.148854 32247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:02:16.150110 32139 server_base.cc:1034] running on GCE node
I20260807 07:02:16.150920 32139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:02:16.152135 32139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:02:16.153409 32139 hybrid_clock.cc:648] HybridClock initialized: now 1786086136153285 us; error 38 us; skew 500 ppm
I20260807 07:02:16.153860 32139 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:02:16 dist-test-slave-ncp9 krb5kdc[30984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086136, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:02:16.158514 32139 init.cc:377] Logged in from keytab as oryx/127.21.83.1@KRBTEST.COM (short username oryx)
I20260807 07:02:16.160600 32139 webserver.cc:460] Webserver started at http://127.21.83.1:45839/ using document root <none> and password file <none>
I20260807 07:02:16.161657 32139 fs_manager.cc:359] Metadata directory not provided
I20260807 07:02:16.161999 32139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:02:16.169294 32139 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.005s
I20260807 07:02:16.172585 32257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:16.173993 32139 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 07:02:16.174181 32139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-0/wal
uuid: "12c53d5fb1164a938c89f771a0d186d4"
format_stamp: "Formatted at 2026-08-07 07:02:12 on dist-test-slave-ncp9"
server_key: "a1934cabac7994b3550b5e0004618cb9"
server_key_iv: "6bea95e64fce2b21939fdd75c8700b57"
server_key_version: "encryptionkey@0"
I20260807 07:02:16.175505 32139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:02:16.193970 32139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:02:16.199476 32139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:02:16.199832 32139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:02:16.201254 32139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:02:16.203366 32139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:02:16.203562 32139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:16.203742 32139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:02:16.203961 32139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:16.290256 32139 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.1:39755
I20260807 07:02:16.290454 32426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.1:39755 every 8 connection(s)
I20260807 07:02:16.291985 32139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
I20260807 07:02:16.297598 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 32139
I20260807 07:02:16.298049 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 31335
Aug 07 07:02:16 dist-test-slave-ncp9 krb5kdc[30984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086136, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.83.1@KRBTEST.COM for oryx/127.21.83.60@KRBTEST.COM
Aug 07 07:02:16 dist-test-slave-ncp9 krb5kdc[30984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086136, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.83.1@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM
Aug 07 07:02:16 dist-test-slave-ncp9 krb5kdc[30984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.83.2@KRBTEST.COM
Aug 07 07:02:16 dist-test-slave-ncp9 krb5kdc[30984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086136, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.83.1@KRBTEST.COM for oryx/127.21.83.61@KRBTEST.COM
I20260807 07:02:16.332760 32243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:16.295695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:33809 (local address 127.21.83.60:43239)
0807 07:02:16.295960 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:16.295967 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:16.302942 (+ 6975us) server_negotiation.cc:407] Connection header received
0807 07:02:16.306755 (+ 3813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:16.306761 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:16.306849 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:16.306951 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:16.310214 (+ 3263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:16.311430 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:16.315350 (+ 3920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:16.316230 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:16.316334 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:16.324258 (+ 7924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:16.324265 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:16.324270 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:16.324311 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:16.328696 (+ 4385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:16.329822 (+ 1126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:16.329829 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:16.329832 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:16.330048 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:16.330721 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:16.330727 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:16.330731 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:16.331090 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:16.331229 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:16.331969 (+ 740us) server_negotiation.cc:299] Negotiation successful
0807 07:02:16.332252 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260807 07:02:16.334113 32435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:16.296269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:43239 (local address 127.21.83.1:33809)
0807 07:02:16.302742 (+ 6473us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:16.302797 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:16.306429 (+ 3632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:16.307537 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:16.307551 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:16.308836 (+ 1285us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:16.309974 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:16.310006 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:16.312241 (+ 2235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:16.312246 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:16.315176 (+ 2930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:16.315192 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:16.316851 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:16.316861 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:16.317265 (+ 404us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:16.318300 (+ 1035us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:16.318337 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:16.323945 (+ 5608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:16.328871 (+ 4926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:16.328882 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:16.328926 (+ 44us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:16.329671 (+ 745us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:16.330222 (+ 551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:16.330227 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:16.330231 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:16.330598 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:16.331229 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:16.331238 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:16.331604 (+ 366us) client_negotiation.cc:769] Sending connection context
0807 07:02:16.331843 (+ 239us) client_negotiation.cc:240] Negotiation successful
0807 07:02:16.332518 (+ 675us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6096,"thread_start_us":12507,"threads_started":1}
I20260807 07:02:16.335971 32428 heartbeater.cc:347] Connected to a master server at 127.21.83.60:43239
I20260807 07:02:16.336375 32428 heartbeater.cc:464] Registering TS with master...
I20260807 07:02:16.336584 32431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:16.295584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:36477 (local address 127.21.83.62:45341)
0807 07:02:16.296709 (+ 1125us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:16.296715 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:16.307944 (+ 11229us) server_negotiation.cc:407] Connection header received
0807 07:02:16.308153 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:16.308158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:16.308231 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:16.308328 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:16.310528 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:16.311765 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:16.317642 (+ 5877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:16.318657 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:16.318791 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:16.325540 (+ 6749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:16.325548 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:16.325552 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:16.325593 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:16.330453 (+ 4860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:16.333111 (+ 2658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:16.333116 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:16.333120 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:16.333334 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:16.334696 (+ 1362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:16.334701 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:16.334704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:16.335107 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:16.335260 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:16.335910 (+ 650us) server_negotiation.cc:299] Negotiation successful
0807 07:02:16.336137 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1009,"thread_start_us":839,"threads_started":1}
I20260807 07:02:16.338023 32428 heartbeater.cc:511] Master 127.21.83.60:43239 requested a full tablet report, sending...
I20260807 07:02:16.339156 32434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:16.297499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:45341 (local address 127.21.83.1:36477)
0807 07:02:16.307847 (+ 10348us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:16.307867 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:16.308038 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:16.308373 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:16.308395 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:16.309791 (+ 1396us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:16.310372 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:16.310386 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:16.311921 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:16.311927 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:16.317489 (+ 5562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:16.317506 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:16.319027 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:16.319036 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:16.319364 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:16.320166 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:16.320190 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:16.324419 (+ 4229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:16.331981 (+ 7562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:16.331987 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:16.331990 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:16.332944 (+ 954us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:16.334301 (+ 1357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:16.334306 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:16.334309 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:16.334575 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:16.335401 (+ 826us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:16.335405 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:16.335561 (+ 156us) client_negotiation.cc:769] Sending connection context
0807 07:02:16.335771 (+ 210us) client_negotiation.cc:240] Negotiation successful
0807 07:02:16.336038 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10156,"mutex_wait_us":2955,"thread_start_us":4629,"threads_started":1}
I20260807 07:02:16.341060 32179 ts_manager.cc:194] Registered new tserver with Master: 12c53d5fb1164a938c89f771a0d186d4 (127.21.83.1:39755)
I20260807 07:02:16.341163 32429 heartbeater.cc:347] Connected to a master server at 127.21.83.62:45341
I20260807 07:02:16.341377 32429 heartbeater.cc:464] Registering TS with master...
I20260807 07:02:16.342128 32429 heartbeater.cc:511] Master 127.21.83.62:45341 requested a full tablet report, sending...
I20260807 07:02:16.342881 32442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:16.309843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47256 (local address 127.21.83.2:45967)
0807 07:02:16.311412 (+ 1569us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:16.311420 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:16.311448 (+ 28us) server_negotiation.cc:407] Connection header received
0807 07:02:16.311511 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:16.311519 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:16.311750 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:16.317722 (+ 5972us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:16.318189 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:16.320006 (+ 1817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:16.321894 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:16.322860 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:16.323652 (+ 792us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:16.330624 (+ 6972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:16.330641 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:16.330657 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:16.330701 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:16.337710 (+ 7009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:16.338796 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:16.338804 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:16.338808 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:16.338977 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:16.339922 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:16.339928 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:16.339931 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:16.340428 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:16.340812 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:16.342095 (+ 1283us) server_negotiation.cc:299] Negotiation successful
0807 07:02:16.342435 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1389,"thread_start_us":7208,"threads_started":1}
I20260807 07:02:16.343724 31048 ts_manager.cc:194] Re-registered known tserver with Master: 12c53d5fb1164a938c89f771a0d186d4 (127.21.83.1:39755)
I20260807 07:02:16.344089 32440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:16.306402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.61:34401 (local address 127.21.83.1:58677)
0807 07:02:16.319371 (+ 12969us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:16.319390 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:16.319525 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:16.320372 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:16.320377 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:16.322448 (+ 2071us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:16.322893 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:16.322907 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:16.325447 (+ 2540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:16.325452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:16.327276 (+ 1824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:16.327292 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:16.328625 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:16.328633 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:16.328853 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:16.329533 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:16.329555 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:16.334097 (+ 4542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:16.340818 (+ 6721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:16.340823 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:16.340827 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:16.341425 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:16.341915 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:16.341921 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:16.341928 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:16.342243 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:16.342993 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:16.342999 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:16.343131 (+ 132us) client_negotiation.cc:769] Sending connection context
0807 07:02:16.343332 (+ 201us) client_negotiation.cc:240] Negotiation successful
0807 07:02:16.343605 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12720,"spinlock_wait_cycles":19200,"thread_start_us":12706,"threads_started":1}
I20260807 07:02:16.344097 32439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:16.305814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:58677 (local address 127.21.83.61:34401)
0807 07:02:16.318704 (+ 12890us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:16.318708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:16.319492 (+ 784us) server_negotiation.cc:407] Connection header received
0807 07:02:16.319650 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:16.319656 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:16.320037 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:16.320185 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:16.324232 (+ 4047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:16.325304 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:16.327457 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:16.328219 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:16.328342 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:16.334899 (+ 6557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:16.334906 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:16.334910 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:16.334952 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:16.340650 (+ 5698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:16.341581 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:16.341586 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:16.341590 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:16.341768 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:16.342384 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:16.342390 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:16.342394 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:16.342854 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:16.343007 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:16.343460 (+ 453us) server_negotiation.cc:299] Negotiation successful
0807 07:02:16.343655 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12783,"thread_start_us":13206,"threads_started":1}
I20260807 07:02:16.344789 31507 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:02:16.345060 31048 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.83.1@KRBTEST.COM'} at 127.21.83.1:36477
I20260807 07:02:16.345587 32430 heartbeater.cc:347] Connected to a master server at 127.21.83.61:34401
I20260807 07:02:16.345832 32430 heartbeater.cc:464] Registering TS with master...
I20260807 07:02:16.346765 32430 heartbeater.cc:511] Master 127.21.83.61:34401 requested a full tablet report, sending...
I20260807 07:02:16.348420 31911 ts_manager.cc:194] Registered new tserver with Master: 12c53d5fb1164a938c89f771a0d186d4 (127.21.83.1:39755)
I20260807 07:02:16.437345 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 31335
I20260807 07:02:16.459560 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.2:45967
--local_ip_for_outbound_sockets=127.21.83.2
--tserver_master_addrs=127.21.83.62:45341
--webserver_port=33513
--webserver_interface=127.21.83.2
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.2.keytab
--principal=oryx/127.21.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:41345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.83.62:45341,127.21.83.61:34401
--tserver_master_addrs=127.21.83.62:45341,127.21.83.61:34401,127.21.83.60:43239 with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 07:02:16.650285 32445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } peers { permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "3ce46c8da37c42dbb13691c7fe185e55" member_type: NON_VOTER last_known_addr { host: "127.21.83.60" port: 43239 } attrs { promote: true } } }
I20260807 07:02:16.650645 32445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb [sys.catalog]: This master's current role is: LEADER
I20260807 07:02:16.659231 32447 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb: attempting to promote NON_VOTER 3ce46c8da37c42dbb13691c7fe185e55 to VOTER
I20260807 07:02:16.660149 32447 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } peers { permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "3ce46c8da37c42dbb13691c7fe185e55" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 43239 } attrs { promote: false } }
I20260807 07:02:16.663014 32189 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55 [term 1 LEARNER]: Refusing update from remote peer 9c117a2eae7c43dd92d88685ac08f1eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260807 07:02:16.663240 31925 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [term 1 FOLLOWER]: Refusing update from remote peer 9c117a2eae7c43dd92d88685ac08f1eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260807 07:02:16.663862 32447 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ce46c8da37c42dbb13691c7fe185e55" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 43239 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260807 07:02:16.664371 32445 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 34401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260807 07:02:16.668033 32447 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3ce46c8da37c42dbb13691c7fe185e55 (127.21.83.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } peers { permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "3ce46c8da37c42dbb13691c7fe185e55" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 43239 } attrs { promote: false } } }
I20260807 07:02:16.668795 31925 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3ce46c8da37c42dbb13691c7fe185e55 (127.21.83.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } peers { permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "3ce46c8da37c42dbb13691c7fe185e55" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 43239 } attrs { promote: false } } }
I20260807 07:02:16.670680 32189 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3ce46c8da37c42dbb13691c7fe185e55 (127.21.83.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } peers { permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "3ce46c8da37c42dbb13691c7fe185e55" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 43239 } attrs { promote: false } } }
I20260807 07:02:16.673141 32446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } peers { permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "3ce46c8da37c42dbb13691c7fe185e55" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 43239 } attrs { promote: false } } }
I20260807 07:02:16.673486 32446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb [sys.catalog]: This master's current role is: LEADER
I20260807 07:02:16.673851 32446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } peers { permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "3ce46c8da37c42dbb13691c7fe185e55" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 43239 } attrs { promote: false } } }
I20260807 07:02:16.674070 32446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb [sys.catalog]: This master's current role is: LEADER
I20260807 07:02:16.674328 32447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } peers { permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "3ce46c8da37c42dbb13691c7fe185e55" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 43239 } attrs { promote: false } } }
I20260807 07:02:16.674551 32447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb [sys.catalog]: This master's current role is: LEADER
I20260807 07:02:16.677356 32458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } peers { permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "3ce46c8da37c42dbb13691c7fe185e55" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 43239 } attrs { promote: false } } }
I20260807 07:02:16.677951 32458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 07:02:16.678476 32454 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } peers { permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "3ce46c8da37c42dbb13691c7fe185e55" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 43239 } attrs { promote: false } } }
I20260807 07:02:16.678885 32454 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [sys.catalog]: This master's current role is: FOLLOWER
W20260807 07:02:16.801539 32444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:02:16.801810 32444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:02:16.801942 32444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:02:16.809347 32444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:02:16.809461 32444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:02:16.809612 32444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.2
I20260807 07:02:16.819249 32444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:41345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.2.keytab
--principal=oryx/127.21.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.2:45967
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.83.2
--webserver_port=33513
--tserver_master_addrs=127.21.83.62:45341,127.21.83.61:34401,127.21.83.60:43239
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.2
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:02:16.820521 32444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:02:16.821861 32444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:02:16.834537 32465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:16.834597 32466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:16.835031 32469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:02:16.835923 32444 server_base.cc:1034] running on GCE node
I20260807 07:02:16.836519 32444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:02:16.837455 32444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:02:16.838660 32444 hybrid_clock.cc:648] HybridClock initialized: now 1786086136838604 us; error 52 us; skew 500 ppm
I20260807 07:02:16.838946 32444 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:02:16 dist-test-slave-ncp9 krb5kdc[30984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086136, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:02:16.842069 32444 init.cc:377] Logged in from keytab as oryx/127.21.83.2@KRBTEST.COM (short username oryx)
I20260807 07:02:16.843466 32444 webserver.cc:460] Webserver started at http://127.21.83.2:33513/ using document root <none> and password file <none>
I20260807 07:02:16.844069 32444 fs_manager.cc:359] Metadata directory not provided
I20260807 07:02:16.844208 32444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:02:16.849385 32444 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260807 07:02:16.851956 32478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:16.853209 32444 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 07:02:16.853354 32444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-1/wal
uuid: "369da2bd392a4341ba2c5af643fc6e84"
format_stamp: "Formatted at 2026-08-07 07:02:13 on dist-test-slave-ncp9"
server_key: "000a5a64ef2d915509f3987156f0da8a"
server_key_iv: "891a28d01dc95d15cb7fb0d7d99059f5"
server_key_version: "encryptionkey@0"
I20260807 07:02:16.854303 32444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:02:16.874023 32444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:02:16.877816 32444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:02:16.878048 32444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:02:16.878944 32444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:02:16.880280 32444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:02:16.880378 32444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:16.880483 32444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:02:16.880558 32444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:16.911100 32444 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.2:45967
I20260807 07:02:16.911170 32634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.2:45967 every 8 connection(s)
I20260807 07:02:16.912485 32444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
I20260807 07:02:16.914384 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 32444
I20260807 07:02:16.914678 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 31523
Aug 07 07:02:16 dist-test-slave-ncp9 krb5kdc[30984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.83.3@KRBTEST.COM
I20260807 07:02:16.960510 32642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:16.915955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35464 (local address 127.21.83.3:42015)
0807 07:02:16.929660 (+ 13705us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:16.929672 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:16.930788 (+ 1116us) server_negotiation.cc:407] Connection header received
0807 07:02:16.930959 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:16.930969 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:16.931255 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:16.931438 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:16.936147 (+ 4709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:16.938046 (+ 1899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:16.940653 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:16.941824 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:16.941927 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:16.950176 (+ 8249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:16.950192 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:16.950208 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:16.950252 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:16.956136 (+ 5884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:16.957214 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:16.957224 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:16.957227 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:16.957482 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:16.958012 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:16.958017 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:16.958020 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:16.958504 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:16.958820 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:16.959610 (+ 790us) server_negotiation.cc:299] Negotiation successful
0807 07:02:16.959900 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13396,"thread_start_us":13723,"threads_started":1}
I20260807 07:02:16.962365 31704 generic_service.cc:196] Checking for leaks (request via RPC)
Aug 07 07:02:16 dist-test-slave-ncp9 krb5kdc[30984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086136, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.83.2@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM
Aug 07 07:02:16 dist-test-slave-ncp9 krb5kdc[30984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086136, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.83.2@KRBTEST.COM for oryx/127.21.83.60@KRBTEST.COM
I20260807 07:02:16.981529 32647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:16.916961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:44561 (local address 127.21.83.62:45341)
0807 07:02:16.934790 (+ 17829us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:16.934797 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:16.936432 (+ 1635us) server_negotiation.cc:407] Connection header received
0807 07:02:16.940987 (+ 4555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:16.940993 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:16.941097 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:16.941199 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:16.944305 (+ 3106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:16.945847 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:16.955646 (+ 9799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:16.956622 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:16.956725 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:16.968864 (+ 12139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:16.968871 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:16.968877 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:16.968977 (+ 100us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:16.974563 (+ 5586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:16.975958 (+ 1395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:16.975965 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:16.975969 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:16.976260 (+ 291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:16.977129 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:16.977134 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:16.977137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:16.977696 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:16.977844 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:16.980250 (+ 2406us) server_negotiation.cc:299] Negotiation successful
0807 07:02:16.980598 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17689,"thread_start_us":18253,"threads_started":1}
Aug 07 07:02:16 dist-test-slave-ncp9 krb5kdc[30984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086136, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.83.2@KRBTEST.COM for oryx/127.21.83.61@KRBTEST.COM
I20260807 07:02:16.985035 32648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:16.916046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:45341 (local address 127.21.83.2:44561)
0807 07:02:16.935824 (+ 19778us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:16.935880 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:16.940651 (+ 4771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:16.941730 (+ 1079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:16.941743 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:16.942837 (+ 1094us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:16.944074 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:16.944100 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:16.946127 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:16.946133 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:16.948839 (+ 2706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:16.948858 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:16.957365 (+ 8507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:16.957381 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:16.957797 (+ 416us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:16.961703 (+ 3906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:16.961735 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:16.968085 (+ 6350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:16.974833 (+ 6748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:16.974846 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:16.974870 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:16.975778 (+ 908us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:16.976440 (+ 662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:16.976445 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:16.976449 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:16.976974 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:16.977975 (+ 1001us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:16.977983 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:16.979058 (+ 1075us) client_negotiation.cc:769] Sending connection context
0807 07:02:16.982130 (+ 3072us) client_negotiation.cc:240] Negotiation successful
0807 07:02:16.982872 (+ 742us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19366,"mutex_wait_us":99,"thread_start_us":20219,"threads_started":1}
I20260807 07:02:16.986600 32646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:16.916371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:52867 (local address 127.21.83.60:43239)
0807 07:02:16.932920 (+ 16549us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:16.932927 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:16.936962 (+ 4035us) server_negotiation.cc:407] Connection header received
0807 07:02:16.949109 (+ 12147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:16.949115 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:16.949224 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:16.949317 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:16.951489 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:16.953002 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:16.959220 (+ 6218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:16.960229 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:16.960328 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:16.976986 (+ 16658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:16.976998 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:16.977003 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:16.977053 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:16.981863 (+ 4810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:16.983463 (+ 1600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:16.983470 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:16.983474 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:16.983689 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:16.984203 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:16.984209 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:16.984212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:16.984685 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:16.984833 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:16.985618 (+ 785us) server_negotiation.cc:299] Negotiation successful
0807 07:02:16.985975 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16406,"thread_start_us":16282,"threads_started":1}
I20260807 07:02:16.987633 32637 heartbeater.cc:347] Connected to a master server at 127.21.83.62:45341
I20260807 07:02:16.988030 32637 heartbeater.cc:464] Registering TS with master...
I20260807 07:02:16.989192 32637 heartbeater.cc:511] Master 127.21.83.62:45341 requested a full tablet report, sending...
I20260807 07:02:16.990553 32644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:16.916046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:43239 (local address 127.21.83.2:52867)
0807 07:02:16.936829 (+ 20783us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:16.936852 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:16.948922 (+ 12070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:16.949615 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:16.949619 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:16.950669 (+ 1050us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:16.951310 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:16.951327 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:16.953242 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:16.953246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:16.955418 (+ 2172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:16.955434 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:16.966397 (+ 10963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:16.966402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:16.966536 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:16.967212 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:16.967235 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:16.976648 (+ 9413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:16.982060 (+ 5412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:16.982065 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:16.982069 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:16.983288 (+ 1219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:16.983866 (+ 578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:16.983870 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:16.983873 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:16.984053 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:16.984965 (+ 912us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:16.984970 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:16.985209 (+ 239us) client_negotiation.cc:769] Sending connection context
0807 07:02:16.986885 (+ 1676us) client_negotiation.cc:240] Negotiation successful
0807 07:02:16.987240 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20572,"mutex_wait_us":739,"thread_start_us":18143,"threads_started":1}
I20260807 07:02:16.991334 31048 ts_manager.cc:194] Re-registered known tserver with Master: 369da2bd392a4341ba2c5af643fc6e84 (127.21.83.2:45967)
I20260807 07:02:16.991689 32636 heartbeater.cc:347] Connected to a master server at 127.21.83.60:43239
I20260807 07:02:16.991863 32636 heartbeater.cc:464] Registering TS with master...
I20260807 07:02:16.992679 32636 heartbeater.cc:511] Master 127.21.83.60:43239 requested a full tablet report, sending...
I20260807 07:02:16.992740 31048 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.83.2@KRBTEST.COM'} at 127.21.83.2:44561
I20260807 07:02:16.994172 32179 ts_manager.cc:194] Registered new tserver with Master: 369da2bd392a4341ba2c5af643fc6e84 (127.21.83.2:45967)
I20260807 07:02:16.995741 32649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:16.937118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:45309 (local address 127.21.83.61:34401)
0807 07:02:16.937995 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:16.938000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:16.961290 (+ 23290us) server_negotiation.cc:407] Connection header received
0807 07:02:16.961453 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:16.961468 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:16.961613 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:16.961716 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:16.966326 (+ 4610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:16.967983 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:16.976856 (+ 8873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:16.977840 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:16.977975 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:16.986368 (+ 8393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:16.986378 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:16.986383 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:16.986431 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:16.991588 (+ 5157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:16.992773 (+ 1185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:16.992780 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:16.992784 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:16.993089 (+ 305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:16.993692 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:16.993698 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:16.993702 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:16.994091 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:16.994261 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:16.994935 (+ 674us) server_negotiation.cc:299] Negotiation successful
0807 07:02:16.995206 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":751,"thread_start_us":606,"threads_started":1}
I20260807 07:02:16.995765 32650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:16.938332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.61:34401 (local address 127.21.83.2:45309)
0807 07:02:16.961179 (+ 22847us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:16.961194 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:16.961297 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:16.964675 (+ 3378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:16.964680 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:16.965671 (+ 991us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:16.966126 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:16.966139 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:16.968218 (+ 2079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:16.968223 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:16.975400 (+ 7177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:16.975423 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:16.978390 (+ 2967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:16.978397 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:16.978791 (+ 394us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:16.979722 (+ 931us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:16.979745 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:16.985956 (+ 6211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:16.991776 (+ 5820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:16.991782 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:16.991787 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:16.992612 (+ 825us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:16.993265 (+ 653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:16.993271 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:16.993276 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:16.993550 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:16.994361 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:16.994367 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:16.994530 (+ 163us) client_negotiation.cc:769] Sending connection context
0807 07:02:16.994709 (+ 179us) client_negotiation.cc:240] Negotiation successful
0807 07:02:16.995066 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22733,"mutex_wait_us":1,"thread_start_us":22495,"threads_started":1}
I20260807 07:02:16.997478 32638 heartbeater.cc:347] Connected to a master server at 127.21.83.61:34401
I20260807 07:02:16.997664 32638 heartbeater.cc:464] Registering TS with master...
I20260807 07:02:16.998605 32638 heartbeater.cc:511] Master 127.21.83.61:34401 requested a full tablet report, sending...
I20260807 07:02:17.000185 31911 ts_manager.cc:194] Registered new tserver with Master: 369da2bd392a4341ba2c5af643fc6e84 (127.21.83.2:45967)
I20260807 07:02:17.065516 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 31523
I20260807 07:02:17.087236 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.3:42015
--local_ip_for_outbound_sockets=127.21.83.3
--tserver_master_addrs=127.21.83.62:45341
--webserver_port=37255
--webserver_interface=127.21.83.3
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.3.keytab
--principal=oryx/127.21.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:41345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.83.62:45341,127.21.83.61:34401
--tserver_master_addrs=127.21.83.62:45341,127.21.83.61:34401,127.21.83.60:43239 with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 07:02:17.348528 32429 heartbeater.cc:503] Master 127.21.83.62:45341 was elected leader, sending a full tablet report...
W20260807 07:02:17.494933 32653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:02:17.495216 32653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:02:17.495318 32653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:02:17.502660 32653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:02:17.502785 32653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:02:17.502912 32653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.3
I20260807 07:02:17.512879 32653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:41345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.3.keytab
--principal=oryx/127.21.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.3:42015
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.83.3
--webserver_port=37255
--tserver_master_addrs=127.21.83.62:45341,127.21.83.61:34401,127.21.83.60:43239
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.3
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:02:17.514321 32653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:02:17.515733 32653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:02:17.528565 32658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:17.528599 32662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:17.528680 32659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:02:17.529752 32653 server_base.cc:1034] running on GCE node
I20260807 07:02:17.530323 32653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:02:17.531219 32653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:02:17.532454 32653 hybrid_clock.cc:648] HybridClock initialized: now 1786086137532398 us; error 42 us; skew 500 ppm
I20260807 07:02:17.532801 32653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:02:17 dist-test-slave-ncp9 krb5kdc[30984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086137, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:02:17.535974 32653 init.cc:377] Logged in from keytab as oryx/127.21.83.3@KRBTEST.COM (short username oryx)
I20260807 07:02:17.537349 32653 webserver.cc:460] Webserver started at http://127.21.83.3:37255/ using document root <none> and password file <none>
I20260807 07:02:17.537988 32653 fs_manager.cc:359] Metadata directory not provided
I20260807 07:02:17.538121 32653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:02:17.543255 32653 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260807 07:02:17.545801 32672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:17.547030 32653 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 07:02:17.547185 32653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-2/wal
uuid: "04c5ff42c0024a9e8f9014904318804e"
format_stamp: "Formatted at 2026-08-07 07:02:13 on dist-test-slave-ncp9"
server_key: "42c9009fb40c4f0b1a7f819855cbce88"
server_key_iv: "1142d1855c5310dd44f404a158a3b428"
server_key_version: "encryptionkey@0"
I20260807 07:02:17.548183 32653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:02:17.569720 32653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:02:17.574751 32653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:02:17.575071 32653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:02:17.576205 32653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:02:17.577809 32653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:02:17.577905 32653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:17.578022 32653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:02:17.578102 32653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:17.610270 32653 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.3:42015
I20260807 07:02:17.610343 355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.3:42015 every 8 connection(s)
I20260807 07:02:17.611720 32653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
I20260807 07:02:17.614538 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 32653
Aug 07 07:02:17 dist-test-slave-ncp9 krb5kdc[30984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086137, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.83.3@KRBTEST.COM for oryx/127.21.83.61@KRBTEST.COM
Aug 07 07:02:17 dist-test-slave-ncp9 krb5kdc[30984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086137, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.83.3@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM
Aug 07 07:02:17 dist-test-slave-ncp9 krb5kdc[30984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086137, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.83.3@KRBTEST.COM for oryx/127.21.83.60@KRBTEST.COM
I20260807 07:02:17.667122 365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:17.615799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:47491 (local address 127.21.83.62:45341)
0807 07:02:17.630452 (+ 14653us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:17.630459 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:17.631101 (+ 642us) server_negotiation.cc:407] Connection header received
0807 07:02:17.636544 (+ 5443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:17.636551 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:17.636644 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:17.636760 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:17.644295 (+ 7535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:17.645718 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:17.647689 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:17.648400 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:17.648546 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:17.655471 (+ 6925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:17.655480 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:17.655508 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:17.655590 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:17.661281 (+ 5691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:17.662854 (+ 1573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:17.662861 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:17.662865 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:17.663170 (+ 305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:17.664115 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:17.664121 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:17.664124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:17.664490 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:17.664654 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:17.666423 (+ 1769us) server_negotiation.cc:299] Negotiation successful
0807 07:02:17.666710 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14521,"thread_start_us":14401,"threads_started":1}
I20260807 07:02:17.667146 369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:17.628204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:46147 (local address 127.21.83.61:34401)
0807 07:02:17.633946 (+ 5742us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:17.633953 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:17.635037 (+ 1084us) server_negotiation.cc:407] Connection header received
0807 07:02:17.636684 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:17.636690 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:17.636785 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:17.638389 (+ 1604us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:17.639561 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:17.640711 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:17.644222 (+ 3511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:17.644973 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:17.645097 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:17.653402 (+ 8305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:17.653410 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:17.653416 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:17.653464 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:17.660416 (+ 6952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:17.662542 (+ 2126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:17.662549 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:17.662552 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:17.662858 (+ 306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:17.663488 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:17.663493 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:17.663496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:17.664011 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:17.664241 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:17.666401 (+ 2160us) server_negotiation.cc:299] Negotiation successful
0807 07:02:17.666686 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5598,"thread_start_us":5488,"threads_started":1}
I20260807 07:02:17.672005 360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:17.614645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:35721 (local address 127.21.83.60:43239)
0807 07:02:17.629927 (+ 15282us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:17.629934 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:17.629970 (+ 36us) server_negotiation.cc:407] Connection header received
0807 07:02:17.637044 (+ 7074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:17.637051 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:17.637146 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:17.637217 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:17.639979 (+ 2762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:17.641374 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:17.656582 (+ 15208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:17.657319 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:17.657430 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:17.661437 (+ 4007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:17.661445 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:17.661449 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:17.661488 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:17.665749 (+ 4261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:17.669294 (+ 3545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:17.669307 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:17.669311 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:17.669521 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:17.670111 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:17.670118 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:17.670122 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:17.670563 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:17.670701 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:17.671300 (+ 599us) server_negotiation.cc:299] Negotiation successful
0807 07:02:17.671571 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15131,"thread_start_us":15325,"threads_started":1}
I20260807 07:02:17.673838 370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:17.628219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.61:34401 (local address 127.21.83.3:46147)
0807 07:02:17.634909 (+ 6690us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:17.634926 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:17.636196 (+ 1270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:17.637102 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:17.637116 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:17.638303 (+ 1187us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:17.639350 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:17.639379 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:17.641588 (+ 2209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:17.641594 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:17.644052 (+ 2458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:17.644072 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:17.646128 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:17.646137 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:17.646489 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:17.647366 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:17.647405 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:17.653109 (+ 5704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:17.661443 (+ 8334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:17.661453 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:17.661473 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:17.662415 (+ 942us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:17.663076 (+ 661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:17.663084 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:17.663088 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:17.663353 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:17.664359 (+ 1006us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:17.664370 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:17.664923 (+ 553us) client_negotiation.cc:769] Sending connection context
0807 07:02:17.665296 (+ 373us) client_negotiation.cc:240] Negotiation successful
0807 07:02:17.665932 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6554,"mutex_wait_us":19,"thread_start_us":6405,"threads_started":1}
I20260807 07:02:17.674352 363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:17.615259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:45341 (local address 127.21.83.3:47491)
0807 07:02:17.630978 (+ 15719us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:17.630999 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:17.636210 (+ 5211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:17.640986 (+ 4776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:17.640993 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:17.641613 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:17.642041 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:17.642054 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:17.645876 (+ 3822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:17.645883 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:17.647537 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:17.647555 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:17.648621 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:17.648627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:17.648812 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:17.649490 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:17.649525 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:17.655140 (+ 5615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:17.662227 (+ 7087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:17.662232 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:17.662236 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:17.662726 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:17.663790 (+ 1064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:17.663795 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:17.663799 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:17.663988 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:17.665901 (+ 1913us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:17.665906 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:17.666068 (+ 162us) client_negotiation.cc:769] Sending connection context
0807 07:02:17.666295 (+ 227us) client_negotiation.cc:240] Negotiation successful
0807 07:02:17.668696 (+ 2401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15577,"mutex_wait_us":35,"thread_start_us":11468,"threads_started":1}
I20260807 07:02:17.674947 364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:17.615259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:43239 (local address 127.21.83.3:35721)
0807 07:02:17.627382 (+ 12123us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:17.627439 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:17.636168 (+ 8729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:17.637469 (+ 1301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:17.637473 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:17.638327 (+ 854us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:17.639791 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:17.639803 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:17.647060 (+ 7257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:17.647066 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:17.654859 (+ 7793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:17.654875 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:17.657730 (+ 2855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:17.657737 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:17.657911 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:17.658463 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:17.658488 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:17.661176 (+ 2688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:17.665867 (+ 4691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:17.665872 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:17.665875 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:17.668461 (+ 2586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:17.669765 (+ 1304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:17.669771 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:17.669775 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:17.669974 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:17.670807 (+ 833us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:17.670813 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:17.670986 (+ 173us) client_negotiation.cc:769] Sending connection context
0807 07:02:17.672939 (+ 1953us) client_negotiation.cc:240] Negotiation successful
0807 07:02:17.673327 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11766,"thread_start_us":15262,"threads_started":1}
I20260807 07:02:17.676040 358 heartbeater.cc:347] Connected to a master server at 127.21.83.61:34401
I20260807 07:02:17.676090 356 heartbeater.cc:347] Connected to a master server at 127.21.83.60:43239
I20260807 07:02:17.676170 357 heartbeater.cc:347] Connected to a master server at 127.21.83.62:45341
I20260807 07:02:17.676465 356 heartbeater.cc:464] Registering TS with master...
I20260807 07:02:17.676460 358 heartbeater.cc:464] Registering TS with master...
I20260807 07:02:17.676460 357 heartbeater.cc:464] Registering TS with master...
I20260807 07:02:17.677721 356 heartbeater.cc:511] Master 127.21.83.60:43239 requested a full tablet report, sending...
I20260807 07:02:17.677791 357 heartbeater.cc:511] Master 127.21.83.62:45341 requested a full tablet report, sending...
I20260807 07:02:17.678122 358 heartbeater.cc:511] Master 127.21.83.61:34401 requested a full tablet report, sending...
I20260807 07:02:17.679970 31048 ts_manager.cc:194] Re-registered known tserver with Master: 04c5ff42c0024a9e8f9014904318804e (127.21.83.3:42015)
I20260807 07:02:17.680042 31911 ts_manager.cc:194] Registered new tserver with Master: 04c5ff42c0024a9e8f9014904318804e (127.21.83.3:42015)
I20260807 07:02:17.681169 31048 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.83.3@KRBTEST.COM'} at 127.21.83.3:47491
I20260807 07:02:17.682512 32179 ts_manager.cc:194] Registered new tserver with Master: 04c5ff42c0024a9e8f9014904318804e (127.21.83.3:42015)
I20260807 07:02:17.688508 373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:17.637509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48548 (local address 127.21.83.60:43239)
0807 07:02:17.656642 (+ 19133us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:17.656648 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:17.656668 (+ 20us) server_negotiation.cc:407] Connection header received
0807 07:02:17.656723 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:17.656728 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:17.656809 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:17.659985 (+ 3176us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:17.660205 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:17.661411 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:17.663394 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:17.670567 (+ 7173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:17.672322 (+ 1755us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:17.678608 (+ 6286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:17.678616 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:17.678620 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:17.678665 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:17.684502 (+ 5837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:17.686457 (+ 1955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:17.686461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:17.686464 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:17.686589 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:17.687001 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:17.687006 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:17.687010 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:17.687365 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:17.687586 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:17.687959 (+ 373us) server_negotiation.cc:299] Negotiation successful
0807 07:02:17.688200 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18960,"thread_start_us":18790,"threads_started":1}
I20260807 07:02:17.692247 365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:17.642626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59034 (local address 127.21.83.62:45341)
0807 07:02:17.667879 (+ 25253us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:17.667885 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:17.667904 (+ 19us) server_negotiation.cc:407] Connection header received
0807 07:02:17.667957 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:17.667962 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:17.668042 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:17.668138 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:17.673433 (+ 5295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:17.674927 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:17.678776 (+ 3849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:17.679694 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:17.679846 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:17.685409 (+ 5563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:17.685415 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:17.685419 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:17.685453 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:17.689354 (+ 3901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:17.690205 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:17.690211 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:17.690215 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:17.690365 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:17.690761 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:17.690767 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:17.690770 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:17.691131 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:17.691325 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:17.691728 (+ 403us) server_negotiation.cc:299] Negotiation successful
0807 07:02:17.691907 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25139,"thread_start_us":28486,"threads_started":1}
I20260807 07:02:17.693243 369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:17.663818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49164 (local address 127.21.83.61:34401)
0807 07:02:17.667531 (+ 3713us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:17.667537 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:17.674000 (+ 6463us) server_negotiation.cc:407] Connection header received
0807 07:02:17.674302 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:17.674309 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:17.674409 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:17.674746 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:17.680225 (+ 5479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:17.681292 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:17.683351 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:17.684019 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:17.684133 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:17.687367 (+ 3234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:17.687372 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:17.687375 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:17.687409 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:17.690385 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:17.691118 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:17.691123 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:17.691125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:17.691282 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:17.691681 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:17.691686 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:17.691689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:17.692025 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:17.692214 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:17.692610 (+ 396us) server_negotiation.cc:299] Negotiation successful
0807 07:02:17.692831 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3603,"thread_start_us":12647,"threads_started":1}
I20260807 07:02:17.696486 31048 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59034:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 07:02:17.703195 31048 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 07:02:17.734266 404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:17.723977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36852 (local address 127.21.83.1:39755)
0807 07:02:17.724597 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:17.724608 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:17.725529 (+ 921us) server_negotiation.cc:407] Connection header received
0807 07:02:17.725762 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:17.725770 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:17.725946 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:17.726127 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:02:17.727099 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:17.728077 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:17.731497 (+ 3420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:17.732450 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:17.732575 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:17.732802 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:17.733872 (+ 1070us) server_negotiation.cc:299] Negotiation successful
0807 07:02:17.734014 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":278,"threads_started":1}
I20260807 07:02:17.735199 401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:17.722567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.3:42015 (local address 127.0.0.1:35480)
0807 07:02:17.723136 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:17.723156 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:17.723284 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:17.726091 (+ 2807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:17.726097 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:17.726136 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:02:17.726637 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:17.726658 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:17.728759 (+ 2101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:17.728765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:17.731302 (+ 2537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:17.731318 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:17.734462 (+ 3144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:17.734468 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:17.734623 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:17.734663 (+ 40us) client_negotiation.cc:769] Sending connection context
0807 07:02:17.734739 (+ 76us) client_negotiation.cc:240] Negotiation successful
0807 07:02:17.734875 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"mutex_wait_us":95,"spinlock_wait_cycles":1280,"thread_start_us":269,"threads_started":1}
I20260807 07:02:17.735564 403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:17.723692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:39755 (local address 127.0.0.1:36852)
0807 07:02:17.724879 (+ 1187us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:17.724934 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:17.725051 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:17.726401 (+ 1350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:17.726405 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:17.726430 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:02:17.726941 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:17.726952 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:17.728234 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:17.728240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:17.731302 (+ 3062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:17.731318 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:17.732611 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:17.732617 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:17.733180 (+ 563us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:17.733216 (+ 36us) client_negotiation.cc:769] Sending connection context
0807 07:02:17.733311 (+ 95us) client_negotiation.cc:240] Negotiation successful
0807 07:02:17.733450 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1063,"mutex_wait_us":68,"thread_start_us":907,"threads_started":1}
I20260807 07:02:17.735749 395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:17.722112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.2:45967 (local address 127.0.0.1:47262)
0807 07:02:17.723812 (+ 1700us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:17.723830 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:17.723948 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:17.726098 (+ 2150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:17.726104 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:17.726150 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:02:17.726759 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:17.726771 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:17.728818 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:17.728828 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:17.730839 (+ 2011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:17.730859 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:17.734843 (+ 3984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:17.734850 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:17.735072 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:17.735105 (+ 33us) client_negotiation.cc:769] Sending connection context
0807 07:02:17.735174 (+ 69us) client_negotiation.cc:240] Negotiation successful
0807 07:02:17.735306 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":32,"spinlock_wait_cycles":1920,"thread_start_us":345,"threads_started":1}
I20260807 07:02:17.736855 399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:17.722386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47262 (local address 127.21.83.2:45967)
0807 07:02:17.725386 (+ 3000us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:17.725394 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:17.725426 (+ 32us) server_negotiation.cc:407] Connection header received
0807 07:02:17.725594 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:17.725601 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:17.725875 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:17.727007 (+ 1132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:02:17.727233 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:17.728655 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:17.731036 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:17.732531 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:17.736150 (+ 3619us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:17.736355 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:17.736416 (+ 61us) server_negotiation.cc:299] Negotiation successful
0807 07:02:17.736591 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2827,"thread_start_us":2875,"threads_started":1}
I20260807 07:02:17.738222 402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:17.723061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35480 (local address 127.21.83.3:42015)
0807 07:02:17.725444 (+ 2383us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:17.725452 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:17.725490 (+ 38us) server_negotiation.cc:407] Connection header received
0807 07:02:17.725593 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:17.725601 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:17.725876 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:17.726085 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:02:17.726887 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:17.728633 (+ 1746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:17.732701 (+ 4068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:17.734271 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:17.734403 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:17.734608 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:17.737740 (+ 3132us) server_negotiation.cc:299] Negotiation successful
0807 07:02:17.737912 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2224,"thread_start_us":2018,"threads_started":1}
I20260807 07:02:17.739673 32332 tablet_service.cc:1467] Processing CreateTablet for tablet 47f6005ecc8f469f9f00556997cc0c88 (DEFAULT_TABLE table=test-table [id=9efa84d6523c40728395c3b7d9a32dde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:02:17.740047 32543 tablet_service.cc:1467] Processing CreateTablet for tablet 47f6005ecc8f469f9f00556997cc0c88 (DEFAULT_TABLE table=test-table [id=9efa84d6523c40728395c3b7d9a32dde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:02:17.741238 32332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47f6005ecc8f469f9f00556997cc0c88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:02:17.741294 32543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47f6005ecc8f469f9f00556997cc0c88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:02:17.741317 32737 tablet_service.cc:1467] Processing CreateTablet for tablet 47f6005ecc8f469f9f00556997cc0c88 (DEFAULT_TABLE table=test-table [id=9efa84d6523c40728395c3b7d9a32dde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:02:17.742559 32737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47f6005ecc8f469f9f00556997cc0c88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:02:17.754703 407 tablet_bootstrap.cc:492] T 47f6005ecc8f469f9f00556997cc0c88 P 04c5ff42c0024a9e8f9014904318804e: Bootstrap starting.
I20260807 07:02:17.756662 409 tablet_bootstrap.cc:492] T 47f6005ecc8f469f9f00556997cc0c88 P 12c53d5fb1164a938c89f771a0d186d4: Bootstrap starting.
I20260807 07:02:17.758301 407 tablet_bootstrap.cc:654] T 47f6005ecc8f469f9f00556997cc0c88 P 04c5ff42c0024a9e8f9014904318804e: Neither blocks nor log segments found. Creating new log.
I20260807 07:02:17.759562 407 log.cc:824] T 47f6005ecc8f469f9f00556997cc0c88 P 04c5ff42c0024a9e8f9014904318804e: Log is configured to *not* fsync() on all Append() calls
I20260807 07:02:17.759941 411 tablet_bootstrap.cc:492] T 47f6005ecc8f469f9f00556997cc0c88 P 369da2bd392a4341ba2c5af643fc6e84: Bootstrap starting.
I20260807 07:02:17.760007 409 tablet_bootstrap.cc:654] T 47f6005ecc8f469f9f00556997cc0c88 P 12c53d5fb1164a938c89f771a0d186d4: Neither blocks nor log segments found. Creating new log.
I20260807 07:02:17.761360 409 log.cc:824] T 47f6005ecc8f469f9f00556997cc0c88 P 12c53d5fb1164a938c89f771a0d186d4: Log is configured to *not* fsync() on all Append() calls
I20260807 07:02:17.763092 411 tablet_bootstrap.cc:654] T 47f6005ecc8f469f9f00556997cc0c88 P 369da2bd392a4341ba2c5af643fc6e84: Neither blocks nor log segments found. Creating new log.
I20260807 07:02:17.764117 411 log.cc:824] T 47f6005ecc8f469f9f00556997cc0c88 P 369da2bd392a4341ba2c5af643fc6e84: Log is configured to *not* fsync() on all Append() calls
I20260807 07:02:17.766150 409 tablet_bootstrap.cc:492] T 47f6005ecc8f469f9f00556997cc0c88 P 12c53d5fb1164a938c89f771a0d186d4: No bootstrap required, opened a new log
I20260807 07:02:17.766549 409 ts_tablet_manager.cc:1397] T 47f6005ecc8f469f9f00556997cc0c88 P 12c53d5fb1164a938c89f771a0d186d4: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260807 07:02:17.770251 407 tablet_bootstrap.cc:492] T 47f6005ecc8f469f9f00556997cc0c88 P 04c5ff42c0024a9e8f9014904318804e: No bootstrap required, opened a new log
I20260807 07:02:17.770665 407 ts_tablet_manager.cc:1397] T 47f6005ecc8f469f9f00556997cc0c88 P 04c5ff42c0024a9e8f9014904318804e: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.008s
I20260807 07:02:17.771030 411 tablet_bootstrap.cc:492] T 47f6005ecc8f469f9f00556997cc0c88 P 369da2bd392a4341ba2c5af643fc6e84: No bootstrap required, opened a new log
I20260807 07:02:17.771370 411 ts_tablet_manager.cc:1397] T 47f6005ecc8f469f9f00556997cc0c88 P 369da2bd392a4341ba2c5af643fc6e84: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260807 07:02:17.780761 409 raft_consensus.cc:348] T 47f6005ecc8f469f9f00556997cc0c88 P 12c53d5fb1164a938c89f771a0d186d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "369da2bd392a4341ba2c5af643fc6e84" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 45967 } } peers { permanent_uuid: "04c5ff42c0024a9e8f9014904318804e" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 42015 } } peers { permanent_uuid: "12c53d5fb1164a938c89f771a0d186d4" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 39755 } }
I20260807 07:02:17.781481 409 raft_consensus.cc:374] T 47f6005ecc8f469f9f00556997cc0c88 P 12c53d5fb1164a938c89f771a0d186d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:02:17.781687 409 raft_consensus.cc:729] T 47f6005ecc8f469f9f00556997cc0c88 P 12c53d5fb1164a938c89f771a0d186d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12c53d5fb1164a938c89f771a0d186d4, State: Initialized, Role: FOLLOWER
I20260807 07:02:17.782634 409 consensus_queue.cc:260] T 47f6005ecc8f469f9f00556997cc0c88 P 12c53d5fb1164a938c89f771a0d186d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "369da2bd392a4341ba2c5af643fc6e84" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 45967 } } peers { permanent_uuid: "04c5ff42c0024a9e8f9014904318804e" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 42015 } } peers { permanent_uuid: "12c53d5fb1164a938c89f771a0d186d4" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 39755 } }
I20260807 07:02:17.785305 409 ts_tablet_manager.cc:1428] T 47f6005ecc8f469f9f00556997cc0c88 P 12c53d5fb1164a938c89f771a0d186d4: Time spent starting tablet: real 0.019s user 0.010s sys 0.008s
I20260807 07:02:17.786509 411 raft_consensus.cc:348] T 47f6005ecc8f469f9f00556997cc0c88 P 369da2bd392a4341ba2c5af643fc6e84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "369da2bd392a4341ba2c5af643fc6e84" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 45967 } } peers { permanent_uuid: "04c5ff42c0024a9e8f9014904318804e" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 42015 } } peers { permanent_uuid: "12c53d5fb1164a938c89f771a0d186d4" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 39755 } }
I20260807 07:02:17.787258 411 raft_consensus.cc:374] T 47f6005ecc8f469f9f00556997cc0c88 P 369da2bd392a4341ba2c5af643fc6e84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:02:17.787490 411 raft_consensus.cc:729] T 47f6005ecc8f469f9f00556997cc0c88 P 369da2bd392a4341ba2c5af643fc6e84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 369da2bd392a4341ba2c5af643fc6e84, State: Initialized, Role: FOLLOWER
I20260807 07:02:17.788395 411 consensus_queue.cc:260] T 47f6005ecc8f469f9f00556997cc0c88 P 369da2bd392a4341ba2c5af643fc6e84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "369da2bd392a4341ba2c5af643fc6e84" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 45967 } } peers { permanent_uuid: "04c5ff42c0024a9e8f9014904318804e" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 42015 } } peers { permanent_uuid: "12c53d5fb1164a938c89f771a0d186d4" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 39755 } }
I20260807 07:02:17.789434 407 raft_consensus.cc:348] T 47f6005ecc8f469f9f00556997cc0c88 P 04c5ff42c0024a9e8f9014904318804e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "369da2bd392a4341ba2c5af643fc6e84" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 45967 } } peers { permanent_uuid: "04c5ff42c0024a9e8f9014904318804e" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 42015 } } peers { permanent_uuid: "12c53d5fb1164a938c89f771a0d186d4" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 39755 } }
I20260807 07:02:17.789898 407 raft_consensus.cc:374] T 47f6005ecc8f469f9f00556997cc0c88 P 04c5ff42c0024a9e8f9014904318804e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:02:17.790016 407 raft_consensus.cc:729] T 47f6005ecc8f469f9f00556997cc0c88 P 04c5ff42c0024a9e8f9014904318804e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04c5ff42c0024a9e8f9014904318804e, State: Initialized, Role: FOLLOWER
I20260807 07:02:17.790591 407 consensus_queue.cc:260] T 47f6005ecc8f469f9f00556997cc0c88 P 04c5ff42c0024a9e8f9014904318804e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "369da2bd392a4341ba2c5af643fc6e84" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 45967 } } peers { permanent_uuid: "04c5ff42c0024a9e8f9014904318804e" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 42015 } } peers { permanent_uuid: "12c53d5fb1164a938c89f771a0d186d4" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 39755 } }
I20260807 07:02:17.797016 357 heartbeater.cc:503] Master 127.21.83.62:45341 was elected leader, sending a full tablet report...
I20260807 07:02:17.797453 407 ts_tablet_manager.cc:1428] T 47f6005ecc8f469f9f00556997cc0c88 P 04c5ff42c0024a9e8f9014904318804e: Time spent starting tablet: real 0.027s user 0.013s sys 0.006s
I20260807 07:02:17.799734 411 ts_tablet_manager.cc:1428] T 47f6005ecc8f469f9f00556997cc0c88 P 369da2bd392a4341ba2c5af643fc6e84: Time spent starting tablet: real 0.028s user 0.009s sys 0.017s
I20260807 07:02:17.799898 32637 heartbeater.cc:503] Master 127.21.83.62:45341 was elected leader, sending a full tablet report...
W20260807 07:02:17.803274 32432 tablet.cc:2369] T 47f6005ecc8f469f9f00556997cc0c88 P 12c53d5fb1164a938c89f771a0d186d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 07:02:17.864283 359 tablet.cc:2369] T 47f6005ecc8f469f9f00556997cc0c88 P 04c5ff42c0024a9e8f9014904318804e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 07:02:17.960047 32639 tablet.cc:2369] T 47f6005ecc8f469f9f00556997cc0c88 P 369da2bd392a4341ba2c5af643fc6e84: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 07:02:18.010524 416 raft_consensus.cc:482] T 47f6005ecc8f469f9f00556997cc0c88 P 12c53d5fb1164a938c89f771a0d186d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:02:18.011003 416 raft_consensus.cc:504] T 47f6005ecc8f469f9f00556997cc0c88 P 12c53d5fb1164a938c89f771a0d186d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "369da2bd392a4341ba2c5af643fc6e84" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 45967 } } peers { permanent_uuid: "04c5ff42c0024a9e8f9014904318804e" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 42015 } } peers { permanent_uuid: "12c53d5fb1164a938c89f771a0d186d4" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 39755 } }
I20260807 07:02:18.013077 416 leader_election.cc:288] T 47f6005ecc8f469f9f00556997cc0c88 P 12c53d5fb1164a938c89f771a0d186d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 369da2bd392a4341ba2c5af643fc6e84 (127.21.83.2:45967), 04c5ff42c0024a9e8f9014904318804e (127.21.83.3:42015)
I20260807 07:02:18.024227 399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:18.013846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:52065 (local address 127.21.83.2:45967)
0807 07:02:18.014085 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:18.014091 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:18.015455 (+ 1364us) server_negotiation.cc:407] Connection header received
0807 07:02:18.015556 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:18.015560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:18.015626 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:18.015755 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:02:18.016776 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:18.017793 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:18.021345 (+ 3552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:18.023001 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:18.023163 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:18.023310 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:18.023835 (+ 525us) server_negotiation.cc:299] Negotiation successful
0807 07:02:18.023969 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260807 07:02:18.024399 402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:18.013945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:58613 (local address 127.21.83.3:42015)
0807 07:02:18.014168 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:18.014174 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:18.015371 (+ 1197us) server_negotiation.cc:407] Connection header received
0807 07:02:18.015959 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:18.015963 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:18.016025 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:18.016094 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:02:18.017400 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:18.018513 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:18.021679 (+ 3166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:18.023106 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:18.023229 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:18.023355 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:18.023997 (+ 642us) server_negotiation.cc:299] Negotiation successful
0807 07:02:18.024126 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260807 07:02:18.024852 419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:18.013349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.3:42015 (local address 127.21.83.1:58613)
0807 07:02:18.015264 (+ 1915us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:18.015279 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:18.015441 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:18.016146 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:18.016151 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:18.016177 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:02:18.017143 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:18.017156 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:18.018725 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:18.018731 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:18.021515 (+ 2784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:18.021527 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:18.023584 (+ 2057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:18.023593 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:18.023825 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:18.023851 (+ 26us) client_negotiation.cc:769] Sending connection context
0807 07:02:18.023939 (+ 88us) client_negotiation.cc:240] Negotiation successful
0807 07:02:18.024073 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1732,"mutex_wait_us":27,"spinlock_wait_cycles":8192,"thread_start_us":1247,"threads_started":1}
I20260807 07:02:18.024866 422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:18.013308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.2:45967 (local address 127.21.83.1:52065)
0807 07:02:18.015246 (+ 1938us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:18.015267 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:18.015441 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:18.015774 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:18.015778 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:18.015801 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:02:18.016613 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:18.016629 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:18.017964 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:18.017968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:18.021173 (+ 3205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:18.021188 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:18.023399 (+ 2211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:18.023409 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:18.023662 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:18.023698 (+ 36us) client_negotiation.cc:769] Sending connection context
0807 07:02:18.023786 (+ 88us) client_negotiation.cc:240] Negotiation successful
0807 07:02:18.023928 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1726,"mutex_wait_us":35,"spinlock_wait_cycles":11904,"thread_start_us":1563,"threads_started":1}
I20260807 07:02:18.026497 32570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47f6005ecc8f469f9f00556997cc0c88" candidate_uuid: "12c53d5fb1164a938c89f771a0d186d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "369da2bd392a4341ba2c5af643fc6e84" is_pre_election: true
I20260807 07:02:18.026473 32764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47f6005ecc8f469f9f00556997cc0c88" candidate_uuid: "12c53d5fb1164a938c89f771a0d186d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04c5ff42c0024a9e8f9014904318804e" is_pre_election: true
I20260807 07:02:18.027314 32570 raft_consensus.cc:2454] T 47f6005ecc8f469f9f00556997cc0c88 P 369da2bd392a4341ba2c5af643fc6e84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12c53d5fb1164a938c89f771a0d186d4 in term 0.
I20260807 07:02:18.027488 32764 raft_consensus.cc:2454] T 47f6005ecc8f469f9f00556997cc0c88 P 04c5ff42c0024a9e8f9014904318804e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12c53d5fb1164a938c89f771a0d186d4 in term 0.
I20260807 07:02:18.028177 32259 leader_election.cc:302] T 47f6005ecc8f469f9f00556997cc0c88 P 12c53d5fb1164a938c89f771a0d186d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12c53d5fb1164a938c89f771a0d186d4, 369da2bd392a4341ba2c5af643fc6e84; no voters:
I20260807 07:02:18.028643 416 raft_consensus.cc:2781] T 47f6005ecc8f469f9f00556997cc0c88 P 12c53d5fb1164a938c89f771a0d186d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 07:02:18.028875 416 raft_consensus.cc:482] T 47f6005ecc8f469f9f00556997cc0c88 P 12c53d5fb1164a938c89f771a0d186d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:02:18.029075 416 raft_consensus.cc:3037] T 47f6005ecc8f469f9f00556997cc0c88 P 12c53d5fb1164a938c89f771a0d186d4 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:02:18.031982 416 raft_consensus.cc:504] T 47f6005ecc8f469f9f00556997cc0c88 P 12c53d5fb1164a938c89f771a0d186d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "369da2bd392a4341ba2c5af643fc6e84" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 45967 } } peers { permanent_uuid: "04c5ff42c0024a9e8f9014904318804e" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 42015 } } peers { permanent_uuid: "12c53d5fb1164a938c89f771a0d186d4" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 39755 } }
I20260807 07:02:18.032727 416 leader_election.cc:288] T 47f6005ecc8f469f9f00556997cc0c88 P 12c53d5fb1164a938c89f771a0d186d4 [CANDIDATE]: Term 1 election: Requested vote from peers 369da2bd392a4341ba2c5af643fc6e84 (127.21.83.2:45967), 04c5ff42c0024a9e8f9014904318804e (127.21.83.3:42015)
I20260807 07:02:18.033267 32764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47f6005ecc8f469f9f00556997cc0c88" candidate_uuid: "12c53d5fb1164a938c89f771a0d186d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04c5ff42c0024a9e8f9014904318804e"
I20260807 07:02:18.033265 32570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47f6005ecc8f469f9f00556997cc0c88" candidate_uuid: "12c53d5fb1164a938c89f771a0d186d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "369da2bd392a4341ba2c5af643fc6e84"
I20260807 07:02:18.033469 32570 raft_consensus.cc:3037] T 47f6005ecc8f469f9f00556997cc0c88 P 369da2bd392a4341ba2c5af643fc6e84 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:02:18.033486 32764 raft_consensus.cc:3037] T 47f6005ecc8f469f9f00556997cc0c88 P 04c5ff42c0024a9e8f9014904318804e [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:02:18.036772 32764 raft_consensus.cc:2454] T 47f6005ecc8f469f9f00556997cc0c88 P 04c5ff42c0024a9e8f9014904318804e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12c53d5fb1164a938c89f771a0d186d4 in term 1.
I20260807 07:02:18.037221 32570 raft_consensus.cc:2454] T 47f6005ecc8f469f9f00556997cc0c88 P 369da2bd392a4341ba2c5af643fc6e84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12c53d5fb1164a938c89f771a0d186d4 in term 1.
I20260807 07:02:18.037261 32261 leader_election.cc:302] T 47f6005ecc8f469f9f00556997cc0c88 P 12c53d5fb1164a938c89f771a0d186d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04c5ff42c0024a9e8f9014904318804e, 12c53d5fb1164a938c89f771a0d186d4; no voters:
I20260807 07:02:18.037616 416 raft_consensus.cc:2781] T 47f6005ecc8f469f9f00556997cc0c88 P 12c53d5fb1164a938c89f771a0d186d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:02:18.038293 416 raft_consensus.cc:686] T 47f6005ecc8f469f9f00556997cc0c88 P 12c53d5fb1164a938c89f771a0d186d4 [term 1 LEADER]: Becoming Leader. State: Replica: 12c53d5fb1164a938c89f771a0d186d4, State: Running, Role: LEADER
I20260807 07:02:18.038926 416 consensus_queue.cc:237] T 47f6005ecc8f469f9f00556997cc0c88 P 12c53d5fb1164a938c89f771a0d186d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "369da2bd392a4341ba2c5af643fc6e84" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 45967 } } peers { permanent_uuid: "04c5ff42c0024a9e8f9014904318804e" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 42015 } } peers { permanent_uuid: "12c53d5fb1164a938c89f771a0d186d4" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 39755 } }
I20260807 07:02:18.044454 31048 catalog_manager.cc:5495] T 47f6005ecc8f469f9f00556997cc0c88 P 12c53d5fb1164a938c89f771a0d186d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12c53d5fb1164a938c89f771a0d186d4 (127.21.83.1). New cstate: current_term: 1 leader_uuid: "12c53d5fb1164a938c89f771a0d186d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "369da2bd392a4341ba2c5af643fc6e84" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 45967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04c5ff42c0024a9e8f9014904318804e" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 42015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12c53d5fb1164a938c89f771a0d186d4" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 39755 } health_report { overall_health: HEALTHY } } }
I20260807 07:02:18.108304 404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:18.101890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36862 (local address 127.21.83.1:39755)
0807 07:02:18.102159 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:18.102166 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:18.102319 (+ 153us) server_negotiation.cc:407] Connection header received
0807 07:02:18.102509 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:18.102512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:18.102623 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:18.102745 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:02:18.103124 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:18.104230 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:18.105913 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:18.106575 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:18.106690 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:18.107192 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:02:18.107717 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:02:18.107819 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:18.107909 (+ 90us) server_negotiation.cc:299] Negotiation successful
0807 07:02:18.107979 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 07:02:18.118705 32764 raft_consensus.cc:1261] T 47f6005ecc8f469f9f00556997cc0c88 P 04c5ff42c0024a9e8f9014904318804e [term 1 FOLLOWER]: Refusing update from remote peer 12c53d5fb1164a938c89f771a0d186d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:02:18.118754 32570 raft_consensus.cc:1261] T 47f6005ecc8f469f9f00556997cc0c88 P 369da2bd392a4341ba2c5af643fc6e84 [term 1 FOLLOWER]: Refusing update from remote peer 12c53d5fb1164a938c89f771a0d186d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:02:18.119871 423 consensus_queue.cc:1035] T 47f6005ecc8f469f9f00556997cc0c88 P 12c53d5fb1164a938c89f771a0d186d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04c5ff42c0024a9e8f9014904318804e" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 42015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 07:02:18.120548 416 consensus_queue.cc:1035] T 47f6005ecc8f469f9f00556997cc0c88 P 12c53d5fb1164a938c89f771a0d186d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "369da2bd392a4341ba2c5af643fc6e84" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 45967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:02:18.137518 426 mvcc.cc:205] Tried to move back new op lower bound from 7315808821722013696 to 7315808821411749888. Current Snapshot: MvccSnapshot[applied={T|T < 7315808821722013696}]
I20260807 07:02:18.159765 31047 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:02:18.160261 31047 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:02:18.170535 31047 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c117a2eae7c43dd92d88685ac08f1eb: Sending DeleteTablet for 3 replicas of tablet 47f6005ecc8f469f9f00556997cc0c88
I20260807 07:02:18.172632 32543 tablet_service.cc:1514] Processing DeleteTablet for tablet 47f6005ecc8f469f9f00556997cc0c88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:02:18 UTC) from {username='oryx', principal='oryx/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:47262
I20260807 07:02:18.174041 32332 tablet_service.cc:1514] Processing DeleteTablet for tablet 47f6005ecc8f469f9f00556997cc0c88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:02:18 UTC) from {username='oryx', principal='oryx/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:36852
I20260807 07:02:18.174335 32737 tablet_service.cc:1514] Processing DeleteTablet for tablet 47f6005ecc8f469f9f00556997cc0c88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:02:18 UTC) from {username='oryx', principal='oryx/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:35480
I20260807 07:02:18.182945 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 31004
I20260807 07:02:18.184620 31094 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:02:18.185597 444 tablet_replica.cc:330] T 47f6005ecc8f469f9f00556997cc0c88 P 04c5ff42c0024a9e8f9014904318804e: stopping tablet replica
I20260807 07:02:18.186409 444 raft_consensus.cc:2229] T 47f6005ecc8f469f9f00556997cc0c88 P 04c5ff42c0024a9e8f9014904318804e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:02:18.187853 444 raft_consensus.cc:2258] T 47f6005ecc8f469f9f00556997cc0c88 P 04c5ff42c0024a9e8f9014904318804e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:02:18.191681 444 ts_tablet_manager.cc:1905] T 47f6005ecc8f469f9f00556997cc0c88 P 04c5ff42c0024a9e8f9014904318804e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:02:18.204046 369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:18.176821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49178 (local address 127.21.83.61:34401)
0807 07:02:18.177026 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:18.177034 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:18.177056 (+ 22us) server_negotiation.cc:407] Connection header received
0807 07:02:18.177127 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:18.177133 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:18.177288 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:18.177411 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:18.180864 (+ 3453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:18.183645 (+ 2781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:18.186374 (+ 2729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:18.187490 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:18.187623 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:18.194131 (+ 6508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:18.194142 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:18.194148 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:18.194218 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:18.199260 (+ 5042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:18.200481 (+ 1221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:18.200488 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:18.200492 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:18.200746 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:18.201394 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:18.201400 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:18.201403 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:18.202046 (+ 643us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:18.202418 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:18.203129 (+ 711us) server_negotiation.cc:299] Negotiation successful
0807 07:02:18.203381 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260807 07:02:18.207329 443 tablet_replica.cc:330] T 47f6005ecc8f469f9f00556997cc0c88 P 369da2bd392a4341ba2c5af643fc6e84: stopping tablet replica
I20260807 07:02:18.207916 443 raft_consensus.cc:2229] T 47f6005ecc8f469f9f00556997cc0c88 P 369da2bd392a4341ba2c5af643fc6e84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:02:18.208680 443 raft_consensus.cc:2258] T 47f6005ecc8f469f9f00556997cc0c88 P 369da2bd392a4341ba2c5af643fc6e84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:02:18.211048 444 ts_tablet_manager.cc:1918] T 47f6005ecc8f469f9f00556997cc0c88 P 04c5ff42c0024a9e8f9014904318804e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:02:18.212817 444 log.cc:1196] T 47f6005ecc8f469f9f00556997cc0c88 P 04c5ff42c0024a9e8f9014904318804e: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-2/wal/wals/47f6005ecc8f469f9f00556997cc0c88
I20260807 07:02:18.213749 441 tablet_replica.cc:330] T 47f6005ecc8f469f9f00556997cc0c88 P 12c53d5fb1164a938c89f771a0d186d4: stopping tablet replica
I20260807 07:02:18.214905 441 raft_consensus.cc:2229] T 47f6005ecc8f469f9f00556997cc0c88 P 12c53d5fb1164a938c89f771a0d186d4 [term 1 LEADER]: Raft consensus shutting down.
I20260807 07:02:18.215482 444 ts_tablet_manager.cc:1939] T 47f6005ecc8f469f9f00556997cc0c88 P 04c5ff42c0024a9e8f9014904318804e: Deleting consensus metadata
I20260807 07:02:18.217324 443 ts_tablet_manager.cc:1905] T 47f6005ecc8f469f9f00556997cc0c88 P 369da2bd392a4341ba2c5af643fc6e84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:02:18.217656 441 raft_consensus.cc:2258] T 47f6005ecc8f469f9f00556997cc0c88 P 12c53d5fb1164a938c89f771a0d186d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:02:18.220093 441 ts_tablet_manager.cc:1905] T 47f6005ecc8f469f9f00556997cc0c88 P 12c53d5fb1164a938c89f771a0d186d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:02:18.229462 373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:18.179761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48560 (local address 127.21.83.60:43239)
0807 07:02:18.179960 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:18.179966 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:18.205549 (+ 25583us) server_negotiation.cc:407] Connection header received
0807 07:02:18.205826 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:18.205833 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:18.205922 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:18.205999 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:18.208953 (+ 2954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:18.210221 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:18.212857 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:18.213722 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:18.213831 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:18.219577 (+ 5746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:18.219588 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:18.219593 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:18.219642 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:18.224730 (+ 5088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:18.226056 (+ 1326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:18.226062 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:18.226066 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:18.226298 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:18.227054 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:18.227059 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:18.227063 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:18.227580 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:18.227864 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:18.228602 (+ 738us) server_negotiation.cc:299] Negotiation successful
0807 07:02:18.228940 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260807 07:02:18.229836 441 ts_tablet_manager.cc:1918] T 47f6005ecc8f469f9f00556997cc0c88 P 12c53d5fb1164a938c89f771a0d186d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:02:18.230202 441 log.cc:1196] T 47f6005ecc8f469f9f00556997cc0c88 P 12c53d5fb1164a938c89f771a0d186d4: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-0/wal/wals/47f6005ecc8f469f9f00556997cc0c88
I20260807 07:02:18.231333 441 ts_tablet_manager.cc:1939] T 47f6005ecc8f469f9f00556997cc0c88 P 12c53d5fb1164a938c89f771a0d186d4: Deleting consensus metadata
I20260807 07:02:18.238241 443 ts_tablet_manager.cc:1918] T 47f6005ecc8f469f9f00556997cc0c88 P 369da2bd392a4341ba2c5af643fc6e84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:02:18.238480 443 log.cc:1196] T 47f6005ecc8f469f9f00556997cc0c88 P 369da2bd392a4341ba2c5af643fc6e84: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-1/wal/wals/47f6005ecc8f469f9f00556997cc0c88
I20260807 07:02:18.239122 443 ts_tablet_manager.cc:1939] T 47f6005ecc8f469f9f00556997cc0c88 P 369da2bd392a4341ba2c5af643fc6e84: Deleting consensus metadata
I20260807 07:02:18.396559 31030 catalog_manager.cc:4841] TS 369da2bd392a4341ba2c5af643fc6e84 (127.21.83.2:45967): tablet 47f6005ecc8f469f9f00556997cc0c88 (table test-table [id=9efa84d6523c40728395c3b7d9a32dde]) successfully deleted
I20260807 07:02:18.396673 31033 catalog_manager.cc:4841] TS 04c5ff42c0024a9e8f9014904318804e (127.21.83.3:42015): tablet 47f6005ecc8f469f9f00556997cc0c88 (table test-table [id=9efa84d6523c40728395c3b7d9a32dde]) successfully deleted
I20260807 07:02:18.397946 31027 catalog_manager.cc:4841] TS 12c53d5fb1164a938c89f771a0d186d4 (127.21.83.1:39755): tablet 47f6005ecc8f469f9f00556997cc0c88 (table test-table [id=9efa84d6523c40728395c3b7d9a32dde]) successfully deleted
I20260807 07:02:18.398145 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 31004
I20260807 07:02:18.448930 373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:18.429715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48572 (local address 127.21.83.60:43239)
0807 07:02:18.429879 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:18.429884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:18.431030 (+ 1146us) server_negotiation.cc:407] Connection header received
0807 07:02:18.431338 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:18.431345 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:18.431452 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:18.431530 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:18.434008 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:18.435234 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:18.437423 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:18.438165 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:18.438270 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:18.442861 (+ 4591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:18.442873 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:18.442879 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:18.442925 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:18.445690 (+ 2765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:18.446534 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:18.446540 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:18.446544 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:18.446732 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:18.447122 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:18.447127 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:18.447131 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:18.447681 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:18.448047 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:18.448315 (+ 268us) server_negotiation.cc:299] Negotiation successful
0807 07:02:18.448554 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260807 07:02:18.461179 369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:18.430744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49182 (local address 127.21.83.61:34401)
0807 07:02:18.430957 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:18.430964 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:18.441540 (+ 10576us) server_negotiation.cc:407] Connection header received
0807 07:02:18.441781 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:18.441787 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:18.441887 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:18.441994 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:18.445943 (+ 3949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:18.446936 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:18.450542 (+ 3606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:18.451002 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:18.451116 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:18.455448 (+ 4332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:18.455454 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:18.455457 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:18.455500 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:18.458229 (+ 2729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:18.459099 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:18.459103 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:18.459106 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:18.459228 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:18.459710 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:18.459714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:18.459716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:18.460038 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:18.460220 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:18.460696 (+ 476us) server_negotiation.cc:299] Negotiation successful
0807 07:02:18.460863 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 07:02:19.145977 455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:19.145277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:45341 (local address 127.21.83.3:38605)
0807 07:02:19.145679 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:19.145798 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:45341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":203,"threads_started":1}
W20260807 07:02:19.146350 357 heartbeater.cc:644] Failed to heartbeat to 127.21.83.62:45341 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.83.62:45341: connect: Connection refused (error 111)
I20260807 07:02:19.149994 456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:19.149291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:45341 (local address 127.21.83.1:55727)
0807 07:02:19.149725 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:19.149835 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:45341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":221,"threads_started":1}
W20260807 07:02:19.150431 32429 heartbeater.cc:644] Failed to heartbeat to 127.21.83.62:45341 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.83.62:45341: connect: Connection refused (error 111)
I20260807 07:02:19.153360 457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:19.152630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:45341 (local address 127.21.83.2:56867)
0807 07:02:19.153058 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:19.153212 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:45341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":185,"threads_started":1}
W20260807 07:02:19.153715 32637 heartbeater.cc:644] Failed to heartbeat to 127.21.83.62:45341 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.83.62:45341: connect: Connection refused (error 111)
I20260807 07:02:19.670846 458 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c117a2eae7c43dd92d88685ac08f1eb)
I20260807 07:02:19.671110 458 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } peers { permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "3ce46c8da37c42dbb13691c7fe185e55" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 43239 } attrs { promote: false } }
I20260807 07:02:19.672452 458 leader_election.cc:288] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c117a2eae7c43dd92d88685ac08f1eb (127.21.83.62:45341), 3ce46c8da37c42dbb13691c7fe185e55 (127.21.83.60:43239)
I20260807 07:02:19.672453 460 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c117a2eae7c43dd92d88685ac08f1eb)
I20260807 07:02:19.672721 460 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } peers { permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "3ce46c8da37c42dbb13691c7fe185e55" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 43239 } attrs { promote: false } }
I20260807 07:02:19.679965 460 leader_election.cc:288] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c117a2eae7c43dd92d88685ac08f1eb (127.21.83.62:45341), a0998212e8024827a4228dd4ca6ed392 (127.21.83.61:34401)
I20260807 07:02:19.680498 461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:19.672675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:45341 (local address 127.0.0.1:57954)
0807 07:02:19.679457 (+ 6782us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:19.680207 (+ 750us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:45341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6591,"thread_start_us":6387,"threads_started":1}
W20260807 07:02:19.680800 31892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.83.62:45341: connect: Connection refused (error 111)
I20260807 07:02:19.682687 461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:19.682282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:45341 (local address 127.0.0.1:57974)
0807 07:02:19.682421 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:19.682510 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:45341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260807 07:02:19.682988 31892 leader_election.cc:334] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9c117a2eae7c43dd92d88685ac08f1eb (127.21.83.62:45341): Network error: Client connection negotiation failed: client connection to 127.21.83.62:45341: connect: Connection refused (error 111)
I20260807 07:02:19.684180 465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:19.673927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:45341 (local address 127.0.0.1:57964)
0807 07:02:19.683747 (+ 9820us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:19.683956 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:45341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8035,"thread_start_us":7872,"threads_started":1}
W20260807 07:02:19.684466 32162 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.83.62:45341: connect: Connection refused (error 111)
I20260807 07:02:19.686201 465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:19.685762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:45341 (local address 127.0.0.1:57982)
0807 07:02:19.685928 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:19.686025 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:45341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260807 07:02:19.687397 32162 leader_election.cc:334] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9c117a2eae7c43dd92d88685ac08f1eb (127.21.83.62:45341): Network error: Client connection negotiation failed: client connection to 127.21.83.62:45341: connect: Connection refused (error 111)
I20260807 07:02:19.687951 464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:19.672970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:43239 (local address 127.0.0.1:52576)
0807 07:02:19.680094 (+ 7124us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:19.680116 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:19.680346 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:19.681246 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:19.681256 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:19.681289 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:02:19.681681 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:19.681694 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:19.683423 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:19.683430 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:19.685814 (+ 2384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:19.685833 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:19.687235 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:19.687240 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:19.687415 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:19.687449 (+ 34us) client_negotiation.cc:769] Sending connection context
0807 07:02:19.687567 (+ 118us) client_negotiation.cc:240] Negotiation successful
0807 07:02:19.687715 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6988,"thread_start_us":6844,"threads_started":1}
I20260807 07:02:19.688144 462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:19.672985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52576 (local address 127.21.83.60:43239)
0807 07:02:19.680710 (+ 7725us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:19.680718 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:19.680740 (+ 22us) server_negotiation.cc:407] Connection header received
0807 07:02:19.680819 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:19.680825 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:19.680925 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:19.681026 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:02:19.681924 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:19.683223 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:19.686016 (+ 2793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:19.687043 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:19.687175 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:19.687274 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:19.687600 (+ 326us) server_negotiation.cc:299] Negotiation successful
0807 07:02:19.687764 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":7575,"thread_start_us":6611,"threads_started":1}
I20260807 07:02:19.688747 32189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a0998212e8024827a4228dd4ca6ed392" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3ce46c8da37c42dbb13691c7fe185e55" is_pre_election: true
I20260807 07:02:19.689160 32189 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0998212e8024827a4228dd4ca6ed392 in term 1.
I20260807 07:02:19.689677 31890 leader_election.cc:302] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ce46c8da37c42dbb13691c7fe185e55, a0998212e8024827a4228dd4ca6ed392; no voters: 9c117a2eae7c43dd92d88685ac08f1eb
I20260807 07:02:19.690040 458 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 07:02:19.690184 458 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9c117a2eae7c43dd92d88685ac08f1eb)
I20260807 07:02:19.690301 458 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [term 1 FOLLOWER]: Advancing to term 2
I20260807 07:02:19.691558 468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:19.682286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.61:34401 (local address 127.0.0.1:35212)
0807 07:02:19.683729 (+ 1443us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:19.683751 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:19.683878 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:19.685924 (+ 2046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:19.685929 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:19.685957 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:02:19.686350 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:19.686362 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:19.687472 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:19.687479 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:19.689442 (+ 1963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:19.689456 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:19.690763 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:19.690770 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:19.690956 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:19.690979 (+ 23us) client_negotiation.cc:769] Sending connection context
0807 07:02:19.691057 (+ 78us) client_negotiation.cc:240] Negotiation successful
0807 07:02:19.691188 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1321,"mutex_wait_us":27,"thread_start_us":1198,"threads_started":1}
I20260807 07:02:19.691632 469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:19.682493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35212 (local address 127.21.83.61:34401)
0807 07:02:19.685627 (+ 3134us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:19.685631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:19.685654 (+ 23us) server_negotiation.cc:407] Connection header received
0807 07:02:19.685707 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:19.685711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:19.685765 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:19.686061 (+ 296us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:02:19.686519 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:19.687323 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:19.689605 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:19.690600 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:19.690720 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:19.690834 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:19.691106 (+ 272us) server_negotiation.cc:299] Negotiation successful
0807 07:02:19.691243 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2996,"thread_start_us":2817,"threads_started":1}
I20260807 07:02:19.692886 31925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3ce46c8da37c42dbb13691c7fe185e55" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a0998212e8024827a4228dd4ca6ed392" is_pre_election: true
I20260807 07:02:19.693377 458 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } peers { permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "3ce46c8da37c42dbb13691c7fe185e55" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 43239 } attrs { promote: false } }
I20260807 07:02:19.693848 31925 raft_consensus.cc:2379] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3ce46c8da37c42dbb13691c7fe185e55 in current term 2: Already voted for candidate a0998212e8024827a4228dd4ca6ed392 in this term.
I20260807 07:02:19.694037 458 leader_election.cc:288] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [CANDIDATE]: Term 2 election: Requested vote from peers 9c117a2eae7c43dd92d88685ac08f1eb (127.21.83.62:45341), 3ce46c8da37c42dbb13691c7fe185e55 (127.21.83.60:43239)
I20260807 07:02:19.694502 32162 leader_election.cc:302] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ce46c8da37c42dbb13691c7fe185e55; no voters: 9c117a2eae7c43dd92d88685ac08f1eb, a0998212e8024827a4228dd4ca6ed392
I20260807 07:02:19.694557 32189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a0998212e8024827a4228dd4ca6ed392" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3ce46c8da37c42dbb13691c7fe185e55"
I20260807 07:02:19.694689 464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:19.694166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:45341 (local address 127.0.0.1:58008)
0807 07:02:19.694467 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:19.694564 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:45341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190}
I20260807 07:02:19.694756 460 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55 [term 1 FOLLOWER]: Advancing to term 2
I20260807 07:02:19.695504 464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:19.695175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:45341 (local address 127.0.0.1:58024)
0807 07:02:19.695312 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:19.695390 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:45341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260807 07:02:19.695652 31892 leader_election.cc:334] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9c117a2eae7c43dd92d88685ac08f1eb (127.21.83.62:45341): Network error: Client connection negotiation failed: client connection to 127.21.83.62:45341: connect: Connection refused (error 111)
I20260807 07:02:19.696780 460 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 07:02:19.698868 32189 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0998212e8024827a4228dd4ca6ed392 in term 2.
I20260807 07:02:19.699890 31890 leader_election.cc:302] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ce46c8da37c42dbb13691c7fe185e55, a0998212e8024827a4228dd4ca6ed392; no voters: 9c117a2eae7c43dd92d88685ac08f1eb
I20260807 07:02:19.700170 458 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 07:02:19.700618 458 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [term 2 LEADER]: Becoming Leader. State: Replica: a0998212e8024827a4228dd4ca6ed392, State: Running, Role: LEADER
I20260807 07:02:19.701000 458 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } peers { permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "3ce46c8da37c42dbb13691c7fe185e55" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 43239 } attrs { promote: false } }
I20260807 07:02:19.702715 473 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0998212e8024827a4228dd4ca6ed392. Latest consensus state: current_term: 2 leader_uuid: "a0998212e8024827a4228dd4ca6ed392" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } peers { permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "3ce46c8da37c42dbb13691c7fe185e55" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 43239 } attrs { promote: false } } }
I20260807 07:02:19.703145 473 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [sys.catalog]: This master's current role is: LEADER
I20260807 07:02:19.703821 476 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:02:19.707351 476 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:02:19.708153 476 catalog_manager.cc:1259] Loaded cluster ID: b2a37eafb25945a195ac09feead69fd5
I20260807 07:02:19.708225 476 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:02:19.709331 476 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:02:19.710105 476 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392: Loaded TSK: 0
I20260807 07:02:19.710863 476 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 07:02:19.751443 31911 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49182:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 07:02:19.759140 31911 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 07:02:19.764501 464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:19.763873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:45341 (local address 127.0.0.1:58042)
0807 07:02:19.764157 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:19.764286 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:45341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
I20260807 07:02:19.765008 32189 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55 [term 2 FOLLOWER]: Refusing update from remote peer a0998212e8024827a4228dd4ca6ed392: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260807 07:02:19.766209 464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:19.765724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:45341 (local address 127.0.0.1:58052)
0807 07:02:19.765935 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:19.766031 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:45341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260807 07:02:19.766279 473 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ce46c8da37c42dbb13691c7fe185e55" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 43239 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260807 07:02:19.766830 31892 consensus_peers.cc:487] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 -> Peer 9c117a2eae7c43dd92d88685ac08f1eb (127.21.83.62:45341): Couldn't send request to peer 9c117a2eae7c43dd92d88685ac08f1eb. Status: Network error: Client connection negotiation failed: client connection to 127.21.83.62:45341: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260807 07:02:19.770478 460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0998212e8024827a4228dd4ca6ed392. Latest consensus state: current_term: 2 leader_uuid: "a0998212e8024827a4228dd4ca6ed392" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } peers { permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "3ce46c8da37c42dbb13691c7fe185e55" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 43239 } attrs { promote: false } } }
I20260807 07:02:19.770820 460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 07:02:19.772349 458 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "a0998212e8024827a4228dd4ca6ed392" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } peers { permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "3ce46c8da37c42dbb13691c7fe185e55" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 43239 } attrs { promote: false } } }
I20260807 07:02:19.772678 458 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392 [sys.catalog]: This master's current role is: LEADER
I20260807 07:02:19.773362 460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "a0998212e8024827a4228dd4ca6ed392" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9c117a2eae7c43dd92d88685ac08f1eb" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 45341 } } peers { permanent_uuid: "a0998212e8024827a4228dd4ca6ed392" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "3ce46c8da37c42dbb13691c7fe185e55" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 43239 } attrs { promote: false } } }
I20260807 07:02:19.773700 460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ce46c8da37c42dbb13691c7fe185e55 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 07:02:19.791638 464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:19.782385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.2:45967 (local address 127.0.0.1:51032)
0807 07:02:19.782531 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:19.782552 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:19.782901 (+ 349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:19.783513 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:19.783518 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:19.783636 (+ 118us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:02:19.783874 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:19.783889 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:19.785971 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:19.785978 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:19.788170 (+ 2192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:19.788201 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:19.790526 (+ 2325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:19.790535 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:19.790795 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:19.790840 (+ 45us) client_negotiation.cc:769] Sending connection context
0807 07:02:19.790960 (+ 120us) client_negotiation.cc:240] Negotiation successful
0807 07:02:19.791272 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260807 07:02:19.793411 485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:19.782524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51032 (local address 127.21.83.2:45967)
0807 07:02:19.783167 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:19.783173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:19.783192 (+ 19us) server_negotiation.cc:407] Connection header received
0807 07:02:19.783264 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:19.783269 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:19.783347 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:19.784385 (+ 1038us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:02:19.784559 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:19.785796 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:19.788412 (+ 2616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:19.790281 (+ 1869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:19.790457 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:19.790653 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:19.790999 (+ 346us) server_negotiation.cc:299] Negotiation successful
0807 07:02:19.793026 (+ 2027us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":381,"threads_started":1}
I20260807 07:02:19.794492 32543 tablet_service.cc:1467] Processing CreateTablet for tablet 46a770ca6b124566829de2b7c4423f4b (DEFAULT_TABLE table=test-table [id=64351fac1e6d4d07ace796af72624d11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:02:19.795070 32543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46a770ca6b124566829de2b7c4423f4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:02:19.802639 489 tablet_bootstrap.cc:492] T 46a770ca6b124566829de2b7c4423f4b P 369da2bd392a4341ba2c5af643fc6e84: Bootstrap starting.
I20260807 07:02:19.804665 461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:19.783502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.3:42015 (local address 127.0.0.1:58770)
0807 07:02:19.784101 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:19.784119 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:19.784242 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:19.797740 (+ 13498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:19.797745 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:19.797781 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:02:19.798126 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:19.798142 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:19.799615 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:19.799621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:19.801750 (+ 2129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:19.801768 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:19.803761 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:19.803769 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:19.803961 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:19.804002 (+ 41us) client_negotiation.cc:769] Sending connection context
0807 07:02:19.804128 (+ 126us) client_negotiation.cc:240] Negotiation successful
0807 07:02:19.804310 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482}
I20260807 07:02:19.805048 489 tablet_bootstrap.cc:654] T 46a770ca6b124566829de2b7c4423f4b P 369da2bd392a4341ba2c5af643fc6e84: Neither blocks nor log segments found. Creating new log.
I20260807 07:02:19.805723 486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:19.783853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58770 (local address 127.21.83.3:42015)
0807 07:02:19.797382 (+ 13529us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:19.797387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:19.797407 (+ 20us) server_negotiation.cc:407] Connection header received
0807 07:02:19.797478 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:19.797481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:19.797546 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:19.797663 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:02:19.798357 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:19.799449 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:19.801965 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:19.803406 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:19.803650 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:19.803815 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:19.805141 (+ 1326us) server_negotiation.cc:299] Negotiation successful
0807 07:02:19.805332 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13406,"thread_start_us":13294,"threads_started":1}
I20260807 07:02:19.806946 32737 tablet_service.cc:1467] Processing CreateTablet for tablet 46a770ca6b124566829de2b7c4423f4b (DEFAULT_TABLE table=test-table [id=64351fac1e6d4d07ace796af72624d11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:02:19.807528 32737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46a770ca6b124566829de2b7c4423f4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:02:19.808243 489 tablet_bootstrap.cc:492] T 46a770ca6b124566829de2b7c4423f4b P 369da2bd392a4341ba2c5af643fc6e84: No bootstrap required, opened a new log
I20260807 07:02:19.808416 489 ts_tablet_manager.cc:1397] T 46a770ca6b124566829de2b7c4423f4b P 369da2bd392a4341ba2c5af643fc6e84: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260807 07:02:19.808670 464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:19.784077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:39755 (local address 127.0.0.1:49362)
0807 07:02:19.791957 (+ 7880us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:19.791981 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:19.792134 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:19.799170 (+ 7036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:19.799175 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:19.799208 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:02:19.799396 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:19.799407 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:19.800651 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:19.800657 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:19.802102 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:19.802117 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:19.807467 (+ 5350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:19.807475 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:19.807883 (+ 408us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:19.807923 (+ 40us) client_negotiation.cc:769] Sending connection context
0807 07:02:19.808039 (+ 116us) client_negotiation.cc:240] Negotiation successful
0807 07:02:19.808215 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7760,"mutex_wait_us":1,"thread_start_us":14823,"threads_started":1}
I20260807 07:02:19.809156 488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:19.784501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49362 (local address 127.21.83.1:39755)
0807 07:02:19.798461 (+ 13960us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:19.798467 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:19.798487 (+ 20us) server_negotiation.cc:407] Connection header received
0807 07:02:19.798559 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:19.798564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:19.798642 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:19.799553 (+ 911us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:02:19.799779 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:19.800479 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:19.804061 (+ 3582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:19.807221 (+ 3160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:19.807405 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:19.807524 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:19.808102 (+ 578us) server_negotiation.cc:299] Negotiation successful
0807 07:02:19.808385 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13813,"thread_start_us":13701,"threads_started":1}
I20260807 07:02:19.810024 32332 tablet_service.cc:1467] Processing CreateTablet for tablet 46a770ca6b124566829de2b7c4423f4b (DEFAULT_TABLE table=test-table [id=64351fac1e6d4d07ace796af72624d11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:02:19.810094 489 raft_consensus.cc:348] T 46a770ca6b124566829de2b7c4423f4b P 369da2bd392a4341ba2c5af643fc6e84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "369da2bd392a4341ba2c5af643fc6e84" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 45967 } } peers { permanent_uuid: "04c5ff42c0024a9e8f9014904318804e" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 42015 } } peers { permanent_uuid: "12c53d5fb1164a938c89f771a0d186d4" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 39755 } }
I20260807 07:02:19.810356 489 raft_consensus.cc:374] T 46a770ca6b124566829de2b7c4423f4b P 369da2bd392a4341ba2c5af643fc6e84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:02:19.810449 489 raft_consensus.cc:729] T 46a770ca6b124566829de2b7c4423f4b P 369da2bd392a4341ba2c5af643fc6e84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 369da2bd392a4341ba2c5af643fc6e84, State: Initialized, Role: FOLLOWER
I20260807 07:02:19.810511 32332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46a770ca6b124566829de2b7c4423f4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:02:19.810843 489 consensus_queue.cc:260] T 46a770ca6b124566829de2b7c4423f4b P 369da2bd392a4341ba2c5af643fc6e84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "369da2bd392a4341ba2c5af643fc6e84" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 45967 } } peers { permanent_uuid: "04c5ff42c0024a9e8f9014904318804e" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 42015 } } peers { permanent_uuid: "12c53d5fb1164a938c89f771a0d186d4" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 39755 } }
I20260807 07:02:19.815119 493 tablet_bootstrap.cc:492] T 46a770ca6b124566829de2b7c4423f4b P 04c5ff42c0024a9e8f9014904318804e: Bootstrap starting.
I20260807 07:02:19.817881 493 tablet_bootstrap.cc:654] T 46a770ca6b124566829de2b7c4423f4b P 04c5ff42c0024a9e8f9014904318804e: Neither blocks nor log segments found. Creating new log.
I20260807 07:02:19.818077 495 tablet_bootstrap.cc:492] T 46a770ca6b124566829de2b7c4423f4b P 12c53d5fb1164a938c89f771a0d186d4: Bootstrap starting.
I20260807 07:02:19.819926 495 tablet_bootstrap.cc:654] T 46a770ca6b124566829de2b7c4423f4b P 12c53d5fb1164a938c89f771a0d186d4: Neither blocks nor log segments found. Creating new log.
I20260807 07:02:19.821259 489 ts_tablet_manager.cc:1428] T 46a770ca6b124566829de2b7c4423f4b P 369da2bd392a4341ba2c5af643fc6e84: Time spent starting tablet: real 0.013s user 0.004s sys 0.004s
I20260807 07:02:19.822297 495 tablet_bootstrap.cc:492] T 46a770ca6b124566829de2b7c4423f4b P 12c53d5fb1164a938c89f771a0d186d4: No bootstrap required, opened a new log
I20260807 07:02:19.822454 496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:19.816838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:45341 (local address 127.21.83.2:40793)
0807 07:02:19.821933 (+ 5095us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:19.822129 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:45341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4926,"threads_started":1}
I20260807 07:02:19.822466 495 ts_tablet_manager.cc:1397] T 46a770ca6b124566829de2b7c4423f4b P 12c53d5fb1164a938c89f771a0d186d4: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260807 07:02:19.823704 495 raft_consensus.cc:348] T 46a770ca6b124566829de2b7c4423f4b P 12c53d5fb1164a938c89f771a0d186d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "369da2bd392a4341ba2c5af643fc6e84" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 45967 } } peers { permanent_uuid: "04c5ff42c0024a9e8f9014904318804e" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 42015 } } peers { permanent_uuid: "12c53d5fb1164a938c89f771a0d186d4" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 39755 } }
I20260807 07:02:19.823958 495 raft_consensus.cc:374] T 46a770ca6b124566829de2b7c4423f4b P 12c53d5fb1164a938c89f771a0d186d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:02:19.824059 495 raft_consensus.cc:729] T 46a770ca6b124566829de2b7c4423f4b P 12c53d5fb1164a938c89f771a0d186d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12c53d5fb1164a938c89f771a0d186d4, State: Initialized, Role: FOLLOWER
I20260807 07:02:19.824453 493 tablet_bootstrap.cc:492] T 46a770ca6b124566829de2b7c4423f4b P 04c5ff42c0024a9e8f9014904318804e: No bootstrap required, opened a new log
I20260807 07:02:19.824355 495 consensus_queue.cc:260] T 46a770ca6b124566829de2b7c4423f4b P 12c53d5fb1164a938c89f771a0d186d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "369da2bd392a4341ba2c5af643fc6e84" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 45967 } } peers { permanent_uuid: "04c5ff42c0024a9e8f9014904318804e" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 42015 } } peers { permanent_uuid: "12c53d5fb1164a938c89f771a0d186d4" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 39755 } }
I20260807 07:02:19.824735 493 ts_tablet_manager.cc:1397] T 46a770ca6b124566829de2b7c4423f4b P 04c5ff42c0024a9e8f9014904318804e: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260807 07:02:19.825909 493 raft_consensus.cc:348] T 46a770ca6b124566829de2b7c4423f4b P 04c5ff42c0024a9e8f9014904318804e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "369da2bd392a4341ba2c5af643fc6e84" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 45967 } } peers { permanent_uuid: "04c5ff42c0024a9e8f9014904318804e" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 42015 } } peers { permanent_uuid: "12c53d5fb1164a938c89f771a0d186d4" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 39755 } }
I20260807 07:02:19.826236 493 raft_consensus.cc:374] T 46a770ca6b124566829de2b7c4423f4b P 04c5ff42c0024a9e8f9014904318804e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:02:19.826236 32638 heartbeater.cc:503] Master 127.21.83.61:34401 was elected leader, sending a full tablet report...
I20260807 07:02:19.826435 496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:19.825584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:45341 (local address 127.21.83.2:39141)
0807 07:02:19.826088 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:19.826185 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:45341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378}
I20260807 07:02:19.826424 493 raft_consensus.cc:729] T 46a770ca6b124566829de2b7c4423f4b P 04c5ff42c0024a9e8f9014904318804e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04c5ff42c0024a9e8f9014904318804e, State: Initialized, Role: FOLLOWER
I20260807 07:02:19.826879 493 consensus_queue.cc:260] T 46a770ca6b124566829de2b7c4423f4b P 04c5ff42c0024a9e8f9014904318804e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "369da2bd392a4341ba2c5af643fc6e84" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 45967 } } peers { permanent_uuid: "04c5ff42c0024a9e8f9014904318804e" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 42015 } } peers { permanent_uuid: "12c53d5fb1164a938c89f771a0d186d4" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 39755 } }
I20260807 07:02:19.827402 495 ts_tablet_manager.cc:1428] T 46a770ca6b124566829de2b7c4423f4b P 12c53d5fb1164a938c89f771a0d186d4: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260807 07:02:19.833877 507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:19.828507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:45341 (local address 127.21.83.1:33987)
0807 07:02:19.833380 (+ 4873us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:19.833571 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:45341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4759,"thread_start_us":4650,"threads_started":1}
I20260807 07:02:19.834127 493 ts_tablet_manager.cc:1428] T 46a770ca6b124566829de2b7c4423f4b P 04c5ff42c0024a9e8f9014904318804e: Time spent starting tablet: real 0.009s user 0.006s sys 0.003s
I20260807 07:02:19.835609 507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:19.835180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:45341 (local address 127.21.83.1:40915)
0807 07:02:19.835356 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:19.835448 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:45341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260807 07:02:19.835815 32430 heartbeater.cc:503] Master 127.21.83.61:34401 was elected leader, sending a full tablet report...
I20260807 07:02:19.836102 508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:19.834756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:45341 (local address 127.21.83.3:40565)
0807 07:02:19.835332 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:19.835459 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:45341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"threads_started":1}
I20260807 07:02:19.837591 358 heartbeater.cc:503] Master 127.21.83.61:34401 was elected leader, sending a full tablet report...
I20260807 07:02:19.838078 508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:19.836653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:45341 (local address 127.21.83.3:47833)
0807 07:02:19.836839 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:19.836935 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:45341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260807 07:02:20.019724 505 raft_consensus.cc:482] T 46a770ca6b124566829de2b7c4423f4b P 04c5ff42c0024a9e8f9014904318804e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:02:20.019966 505 raft_consensus.cc:504] T 46a770ca6b124566829de2b7c4423f4b P 04c5ff42c0024a9e8f9014904318804e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "369da2bd392a4341ba2c5af643fc6e84" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 45967 } } peers { permanent_uuid: "04c5ff42c0024a9e8f9014904318804e" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 42015 } } peers { permanent_uuid: "12c53d5fb1164a938c89f771a0d186d4" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 39755 } }
I20260807 07:02:20.021646 505 leader_election.cc:288] T 46a770ca6b124566829de2b7c4423f4b P 04c5ff42c0024a9e8f9014904318804e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 369da2bd392a4341ba2c5af643fc6e84 (127.21.83.2:45967), 12c53d5fb1164a938c89f771a0d186d4 (127.21.83.1:39755)
I20260807 07:02:20.029121 485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:20.022011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:58857 (local address 127.21.83.2:45967)
0807 07:02:20.022274 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:20.022280 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:20.022306 (+ 26us) server_negotiation.cc:407] Connection header received
0807 07:02:20.022542 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:20.022548 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:20.022632 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:20.022712 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:02:20.023204 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:20.024258 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:20.026773 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:20.028020 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:20.028141 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:20.028246 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:20.028683 (+ 437us) server_negotiation.cc:299] Negotiation successful
0807 07:02:20.028822 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 07:02:20.029121 488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:20.022135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:49847 (local address 127.21.83.1:39755)
0807 07:02:20.022378 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:20.022383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:20.022753 (+ 370us) server_negotiation.cc:407] Connection header received
0807 07:02:20.022900 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:20.022904 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:20.022976 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:20.023048 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:02:20.023501 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:20.024490 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:20.026761 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:20.027987 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:20.028127 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:20.028227 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:20.028691 (+ 464us) server_negotiation.cc:299] Negotiation successful
0807 07:02:20.028822 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260807 07:02:20.029217 508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:20.021872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.2:45967 (local address 127.21.83.3:58857)
0807 07:02:20.022161 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:20.022183 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:20.022351 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:20.022762 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:20.022767 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:20.022808 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:02:20.023043 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:20.023055 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:20.024386 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:20.024389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:20.026577 (+ 2188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:20.026593 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:20.028209 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:20.028214 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:20.028438 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:20.028553 (+ 115us) client_negotiation.cc:769] Sending connection context
0807 07:02:20.028653 (+ 100us) client_negotiation.cc:240] Negotiation successful
0807 07:02:20.028835 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":83}
I20260807 07:02:20.029713 509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:20.022008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:39755 (local address 127.21.83.3:49847)
0807 07:02:20.022648 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:20.022661 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:20.022775 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:20.023089 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:20.023093 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:20.023118 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:02:20.023368 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:20.023376 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:20.024609 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:20.024614 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:20.026611 (+ 1997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:20.026621 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:20.028189 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:20.028195 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:20.028490 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:20.028558 (+ 68us) client_negotiation.cc:769] Sending connection context
0807 07:02:20.028658 (+ 100us) client_negotiation.cc:240] Negotiation successful
0807 07:02:20.028828 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"mutex_wait_us":107,"thread_start_us":372,"threads_started":1}
I20260807 07:02:20.029829 32570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46a770ca6b124566829de2b7c4423f4b" candidate_uuid: "04c5ff42c0024a9e8f9014904318804e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "369da2bd392a4341ba2c5af643fc6e84" is_pre_election: true
I20260807 07:02:20.030130 32570 raft_consensus.cc:2454] T 46a770ca6b124566829de2b7c4423f4b P 369da2bd392a4341ba2c5af643fc6e84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04c5ff42c0024a9e8f9014904318804e in term 0.
I20260807 07:02:20.030568 32364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46a770ca6b124566829de2b7c4423f4b" candidate_uuid: "04c5ff42c0024a9e8f9014904318804e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12c53d5fb1164a938c89f771a0d186d4" is_pre_election: true
I20260807 07:02:20.030702 32674 leader_election.cc:302] T 46a770ca6b124566829de2b7c4423f4b P 04c5ff42c0024a9e8f9014904318804e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04c5ff42c0024a9e8f9014904318804e, 369da2bd392a4341ba2c5af643fc6e84; no voters:
I20260807 07:02:20.030930 32364 raft_consensus.cc:2454] T 46a770ca6b124566829de2b7c4423f4b P 12c53d5fb1164a938c89f771a0d186d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04c5ff42c0024a9e8f9014904318804e in term 0.
I20260807 07:02:20.031128 505 raft_consensus.cc:2781] T 46a770ca6b124566829de2b7c4423f4b P 04c5ff42c0024a9e8f9014904318804e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 07:02:20.031260 505 raft_consensus.cc:482] T 46a770ca6b124566829de2b7c4423f4b P 04c5ff42c0024a9e8f9014904318804e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:02:20.031365 505 raft_consensus.cc:3037] T 46a770ca6b124566829de2b7c4423f4b P 04c5ff42c0024a9e8f9014904318804e [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:02:20.033667 505 raft_consensus.cc:504] T 46a770ca6b124566829de2b7c4423f4b P 04c5ff42c0024a9e8f9014904318804e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "369da2bd392a4341ba2c5af643fc6e84" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 45967 } } peers { permanent_uuid: "04c5ff42c0024a9e8f9014904318804e" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 42015 } } peers { permanent_uuid: "12c53d5fb1164a938c89f771a0d186d4" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 39755 } }
I20260807 07:02:20.034267 505 leader_election.cc:288] T 46a770ca6b124566829de2b7c4423f4b P 04c5ff42c0024a9e8f9014904318804e [CANDIDATE]: Term 1 election: Requested vote from peers 369da2bd392a4341ba2c5af643fc6e84 (127.21.83.2:45967), 12c53d5fb1164a938c89f771a0d186d4 (127.21.83.1:39755)
I20260807 07:02:20.034807 32364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46a770ca6b124566829de2b7c4423f4b" candidate_uuid: "04c5ff42c0024a9e8f9014904318804e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12c53d5fb1164a938c89f771a0d186d4"
I20260807 07:02:20.034783 32570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46a770ca6b124566829de2b7c4423f4b" candidate_uuid: "04c5ff42c0024a9e8f9014904318804e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "369da2bd392a4341ba2c5af643fc6e84"
I20260807 07:02:20.035008 32364 raft_consensus.cc:3037] T 46a770ca6b124566829de2b7c4423f4b P 12c53d5fb1164a938c89f771a0d186d4 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:02:20.035030 32570 raft_consensus.cc:3037] T 46a770ca6b124566829de2b7c4423f4b P 369da2bd392a4341ba2c5af643fc6e84 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:02:20.038064 32570 raft_consensus.cc:2454] T 46a770ca6b124566829de2b7c4423f4b P 369da2bd392a4341ba2c5af643fc6e84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04c5ff42c0024a9e8f9014904318804e in term 1.
I20260807 07:02:20.038065 32364 raft_consensus.cc:2454] T 46a770ca6b124566829de2b7c4423f4b P 12c53d5fb1164a938c89f771a0d186d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04c5ff42c0024a9e8f9014904318804e in term 1.
I20260807 07:02:20.038496 32673 leader_election.cc:302] T 46a770ca6b124566829de2b7c4423f4b P 04c5ff42c0024a9e8f9014904318804e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04c5ff42c0024a9e8f9014904318804e, 12c53d5fb1164a938c89f771a0d186d4; no voters:
I20260807 07:02:20.038841 505 raft_consensus.cc:2781] T 46a770ca6b124566829de2b7c4423f4b P 04c5ff42c0024a9e8f9014904318804e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:02:20.039289 505 raft_consensus.cc:686] T 46a770ca6b124566829de2b7c4423f4b P 04c5ff42c0024a9e8f9014904318804e [term 1 LEADER]: Becoming Leader. State: Replica: 04c5ff42c0024a9e8f9014904318804e, State: Running, Role: LEADER
I20260807 07:02:20.039919 505 consensus_queue.cc:237] T 46a770ca6b124566829de2b7c4423f4b P 04c5ff42c0024a9e8f9014904318804e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "369da2bd392a4341ba2c5af643fc6e84" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 45967 } } peers { permanent_uuid: "04c5ff42c0024a9e8f9014904318804e" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 42015 } } peers { permanent_uuid: "12c53d5fb1164a938c89f771a0d186d4" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 39755 } }
I20260807 07:02:20.040325 509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:20.039907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:45341 (local address 127.21.83.3:38457)
0807 07:02:20.040054 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:20.040161 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:45341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260807 07:02:20.043804 31909 catalog_manager.cc:5495] T 46a770ca6b124566829de2b7c4423f4b P 04c5ff42c0024a9e8f9014904318804e reported cstate change: term changed from 0 to 1, leader changed from <none> to 04c5ff42c0024a9e8f9014904318804e (127.21.83.3). New cstate: current_term: 1 leader_uuid: "04c5ff42c0024a9e8f9014904318804e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "369da2bd392a4341ba2c5af643fc6e84" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 45967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04c5ff42c0024a9e8f9014904318804e" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 42015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12c53d5fb1164a938c89f771a0d186d4" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 39755 } health_report { overall_health: UNKNOWN } } }
I20260807 07:02:20.106760 486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:20.101466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58780 (local address 127.21.83.3:42015)
0807 07:02:20.101675 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:20.101693 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:20.101713 (+ 20us) server_negotiation.cc:407] Connection header received
0807 07:02:20.101915 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:20.101919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:20.102007 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:20.102143 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:02:20.102530 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:20.103255 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:20.104564 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:20.105038 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:20.105154 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:20.105563 (+ 409us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:02:20.106128 (+ 565us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:02:20.106261 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:20.106405 (+ 144us) server_negotiation.cc:299] Negotiation successful
0807 07:02:20.106489 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 07:02:20.110913 32570 raft_consensus.cc:1261] T 46a770ca6b124566829de2b7c4423f4b P 369da2bd392a4341ba2c5af643fc6e84 [term 1 FOLLOWER]: Refusing update from remote peer 04c5ff42c0024a9e8f9014904318804e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:02:20.111255 32364 raft_consensus.cc:1261] T 46a770ca6b124566829de2b7c4423f4b P 12c53d5fb1164a938c89f771a0d186d4 [term 1 FOLLOWER]: Refusing update from remote peer 04c5ff42c0024a9e8f9014904318804e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:02:20.111680 510 consensus_queue.cc:1035] T 46a770ca6b124566829de2b7c4423f4b P 04c5ff42c0024a9e8f9014904318804e [LEADER]: Connected to new peer: Peer: permanent_uuid: "369da2bd392a4341ba2c5af643fc6e84" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 45967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:02:20.112057 505 consensus_queue.cc:1035] T 46a770ca6b124566829de2b7c4423f4b P 04c5ff42c0024a9e8f9014904318804e [LEADER]: Connected to new peer: Peer: permanent_uuid: "12c53d5fb1164a938c89f771a0d186d4" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 39755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:02:20.114553 507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:20.114056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:45341 (local address 127.21.83.1:44023)
0807 07:02:20.114246 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:20.114370 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:45341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260807 07:02:20.115371 496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:20.114824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:45341 (local address 127.21.83.2:38893)
0807 07:02:20.115045 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:20.115124 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:45341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260807 07:02:20.121551 496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:20.121065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:45341 (local address 127.21.83.2:48003)
0807 07:02:20.121263 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:20.121368 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:45341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260807 07:02:20.123119 509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:20.122546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:45341 (local address 127.21.83.3:48749)
0807 07:02:20.122799 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:20.122927 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:45341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260807 07:02:20.127358 509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:20.126911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:45341 (local address 127.21.83.3:51337)
0807 07:02:20.127079 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:20.127177 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:45341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260807 07:02:20.128939 507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:20.128445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:45341 (local address 127.21.83.1:44557)
0807 07:02:20.128673 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:20.128771 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:45341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260807 07:02:20.137291 31909 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:02:20.137509 31909 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:02:20.142711 31909 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a0998212e8024827a4228dd4ca6ed392: Sending DeleteTablet for 3 replicas of tablet 46a770ca6b124566829de2b7c4423f4b
I20260807 07:02:20.143549 32543 tablet_service.cc:1514] Processing DeleteTablet for tablet 46a770ca6b124566829de2b7c4423f4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:02:20 UTC) from {username='oryx', principal='oryx/127.21.83.61@KRBTEST.COM'} at 127.0.0.1:51032
I20260807 07:02:20.143565 32737 tablet_service.cc:1514] Processing DeleteTablet for tablet 46a770ca6b124566829de2b7c4423f4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:02:20 UTC) from {username='oryx', principal='oryx/127.21.83.61@KRBTEST.COM'} at 127.0.0.1:58770
I20260807 07:02:20.145769 32332 tablet_service.cc:1514] Processing DeleteTablet for tablet 46a770ca6b124566829de2b7c4423f4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:02:20 UTC) from {username='oryx', principal='oryx/127.21.83.61@KRBTEST.COM'} at 127.0.0.1:49362
I20260807 07:02:20.145877 529 tablet_replica.cc:330] T 46a770ca6b124566829de2b7c4423f4b P 369da2bd392a4341ba2c5af643fc6e84: stopping tablet replica
I20260807 07:02:20.146075 529 raft_consensus.cc:2229] T 46a770ca6b124566829de2b7c4423f4b P 369da2bd392a4341ba2c5af643fc6e84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:02:20.146461 531 tablet_replica.cc:330] T 46a770ca6b124566829de2b7c4423f4b P 12c53d5fb1164a938c89f771a0d186d4: stopping tablet replica
I20260807 07:02:20.146715 531 raft_consensus.cc:2229] T 46a770ca6b124566829de2b7c4423f4b P 12c53d5fb1164a938c89f771a0d186d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:02:20.146412 529 raft_consensus.cc:2258] T 46a770ca6b124566829de2b7c4423f4b P 369da2bd392a4341ba2c5af643fc6e84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:02:20.147341 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 32139
I20260807 07:02:20.147382 530 tablet_replica.cc:330] T 46a770ca6b124566829de2b7c4423f4b P 04c5ff42c0024a9e8f9014904318804e: stopping tablet replica
I20260807 07:02:20.147567 530 raft_consensus.cc:2229] T 46a770ca6b124566829de2b7c4423f4b P 04c5ff42c0024a9e8f9014904318804e [term 1 LEADER]: Raft consensus shutting down.
I20260807 07:02:20.148162 529 ts_tablet_manager.cc:1905] T 46a770ca6b124566829de2b7c4423f4b P 369da2bd392a4341ba2c5af643fc6e84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:02:20.148438 531 raft_consensus.cc:2258] T 46a770ca6b124566829de2b7c4423f4b P 12c53d5fb1164a938c89f771a0d186d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:02:20.149819 531 ts_tablet_manager.cc:1905] T 46a770ca6b124566829de2b7c4423f4b P 12c53d5fb1164a938c89f771a0d186d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:02:20.150731 530 raft_consensus.cc:2258] T 46a770ca6b124566829de2b7c4423f4b P 04c5ff42c0024a9e8f9014904318804e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:02:20.151788 530 ts_tablet_manager.cc:1905] T 46a770ca6b124566829de2b7c4423f4b P 04c5ff42c0024a9e8f9014904318804e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:02:20.158927 530 ts_tablet_manager.cc:1918] T 46a770ca6b124566829de2b7c4423f4b P 04c5ff42c0024a9e8f9014904318804e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:02:20.159111 530 log.cc:1196] T 46a770ca6b124566829de2b7c4423f4b P 04c5ff42c0024a9e8f9014904318804e: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-2/wal/wals/46a770ca6b124566829de2b7c4423f4b
I20260807 07:02:20.159658 530 ts_tablet_manager.cc:1939] T 46a770ca6b124566829de2b7c4423f4b P 04c5ff42c0024a9e8f9014904318804e: Deleting consensus metadata
I20260807 07:02:20.161469 31892 catalog_manager.cc:4841] TS 04c5ff42c0024a9e8f9014904318804e (127.21.83.3:42015): tablet 46a770ca6b124566829de2b7c4423f4b (table test-table [id=64351fac1e6d4d07ace796af72624d11]) successfully deleted
I20260807 07:02:20.163295 531 ts_tablet_manager.cc:1918] T 46a770ca6b124566829de2b7c4423f4b P 12c53d5fb1164a938c89f771a0d186d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:02:20.163450 531 log.cc:1196] T 46a770ca6b124566829de2b7c4423f4b P 12c53d5fb1164a938c89f771a0d186d4: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-0/wal/wals/46a770ca6b124566829de2b7c4423f4b
I20260807 07:02:20.163872 531 ts_tablet_manager.cc:1939] T 46a770ca6b124566829de2b7c4423f4b P 12c53d5fb1164a938c89f771a0d186d4: Deleting consensus metadata
I20260807 07:02:20.165114 31889 catalog_manager.cc:4841] TS 12c53d5fb1164a938c89f771a0d186d4 (127.21.83.1:39755): tablet 46a770ca6b124566829de2b7c4423f4b (table test-table [id=64351fac1e6d4d07ace796af72624d11]) successfully deleted
I20260807 07:02:20.165649 529 ts_tablet_manager.cc:1918] T 46a770ca6b124566829de2b7c4423f4b P 369da2bd392a4341ba2c5af643fc6e84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:02:20.165798 529 log.cc:1196] T 46a770ca6b124566829de2b7c4423f4b P 369da2bd392a4341ba2c5af643fc6e84: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786086078080656-21836-0/minicluster-data/ts-1/wal/wals/46a770ca6b124566829de2b7c4423f4b
I20260807 07:02:20.166266 529 ts_tablet_manager.cc:1939] T 46a770ca6b124566829de2b7c4423f4b P 369da2bd392a4341ba2c5af643fc6e84: Deleting consensus metadata
I20260807 07:02:20.167544 31890 catalog_manager.cc:4841] TS 369da2bd392a4341ba2c5af643fc6e84 (127.21.83.2:45967): tablet 46a770ca6b124566829de2b7c4423f4b (table test-table [id=64351fac1e6d4d07ace796af72624d11]) successfully deleted
I20260807 07:02:20.169247 488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:20.148649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49364 (local address 127.21.83.1:39755)
0807 07:02:20.148802 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:20.148807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:20.150387 (+ 1580us) server_negotiation.cc:407] Connection header received
0807 07:02:20.150627 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:20.150632 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:20.150714 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:20.150780 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:20.152743 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:20.153860 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:20.155604 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:20.156237 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:20.156336 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:20.160207 (+ 3871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:20.160229 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:20.160250 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:20.160304 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:20.164954 (+ 4650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:20.166126 (+ 1172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:20.166138 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:20.166141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:20.166346 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:20.166877 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:20.166883 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:20.166886 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:20.167521 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:20.167937 (+ 416us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:20.168568 (+ 631us) server_negotiation.cc:299] Negotiation successful
0807 07:02:20.168850 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 07:02:20.169901 32421 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:02:20.218250 464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:20.217751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:45341 (local address 127.0.0.1:58056)
0807 07:02:20.217932 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:20.218041 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:45341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260807 07:02:20.219215 464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:20.218878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:45341 (local address 127.0.0.1:58070)
0807 07:02:20.219021 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:20.219094 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:45341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260807 07:02:20.342331 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 32139
I20260807 07:02:20.372442 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 32444
I20260807 07:02:20.388765 485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:20.373245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51034 (local address 127.21.83.2:45967)
0807 07:02:20.373481 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:20.373488 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:20.373511 (+ 23us) server_negotiation.cc:407] Connection header received
0807 07:02:20.373648 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:20.373653 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:20.373733 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:20.373829 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:20.375169 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:20.376375 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:20.377957 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:20.378566 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:20.378720 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:20.381506 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:20.381529 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:20.381549 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:20.381610 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:20.385533 (+ 3923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:20.386342 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:20.386354 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:20.386357 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:20.386555 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:20.387007 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:20.387012 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:20.387015 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:20.387535 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:20.387869 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:20.388100 (+ 231us) server_negotiation.cc:299] Negotiation successful
0807 07:02:20.388373 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260807 07:02:20.389590 32629 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:02:20.486008 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 32444
I20260807 07:02:20.508759 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 32653
I20260807 07:02:20.524274 486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:20.509739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58792 (local address 127.21.83.3:42015)
0807 07:02:20.509947 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:20.509953 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:20.509975 (+ 22us) server_negotiation.cc:407] Connection header received
0807 07:02:20.510043 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:20.510049 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:20.510130 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:20.510247 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:20.511500 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:20.512693 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:20.514005 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:20.514660 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:20.514808 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:20.517235 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:20.517252 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:20.517271 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:20.517314 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:20.521385 (+ 4071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:20.522012 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:20.522021 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:20.522025 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:20.522264 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:20.522603 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:20.522609 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:20.522613 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:20.523102 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:20.523489 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:20.523658 (+ 169us) server_negotiation.cc:299] Negotiation successful
0807 07:02:20.523901 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 07:02:20.525285 348 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:02:20.656863 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 32653
I20260807 07:02:20.686653 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 31721
I20260807 07:02:20.687870 31952 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:02:20.810571 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 31721
W20260807 07:02:20.832196 32157 connection.cc:535] server connection from 127.0.0.1:52576 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260807 07:02:20.832551 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 31867
I20260807 07:02:20.833485 32213 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:02:20.952939 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 31867
2026-08-07T07:02:20Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10892 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 07:02:21 dist-test-slave-ncp9 krb5kdc[544](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 07:02:21 dist-test-slave-ncp9 krb5kdc[544](info): set up 2 sockets
Aug 07 07:02:21 dist-test-slave-ncp9 krb5kdc[544](info): commencing operation
krb5kdc: starting...
W20260807 07:02:22.185838 21836 mini_kdc.cc:121] Time spent starting KDC: real 1.180s user 0.001s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Aug 07 07:02:22 dist-test-slave-ncp9 krb5kdc[544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T07:02:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:02:22Z Disabled control of system clock
WARNING: no policy specified for oryx/127.21.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.62.keytab.
Entry for principal oryx/127.21.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.62.keytab.
I20260807 07:02:22.450460 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:35983
--webserver_interface=127.21.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.62.keytab
--principal=oryx/127.21.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:41903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:35983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:02:22.783005 562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:02:22.783281 562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:02:22.783380 562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:02:22.790376 562 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:02:22.790467 562 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260807 07:02:22.790514 562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:02:22.790575 562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:02:22.790630 562 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:02:22.790690 562 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:02:22.800005 562 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:41903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:35983
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.62.keytab
--principal=oryx/127.21.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:35983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:02:22.801187 562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:02:22.802417 562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:02:22.813149 569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:22.813247 570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:22.813217 573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:02:22.814452 562 server_base.cc:1034] running on GCE node
I20260807 07:02:22.815181 562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:02:22.816466 562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:02:22.817679 562 hybrid_clock.cc:648] HybridClock initialized: now 1786086142817608 us; error 46 us; skew 500 ppm
I20260807 07:02:22.817968 562 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:02:22 dist-test-slave-ncp9 krb5kdc[544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086142, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:02:22.820999 562 init.cc:377] Logged in from keytab as oryx/127.21.83.62@KRBTEST.COM (short username oryx)
I20260807 07:02:22.822355 562 webserver.cc:460] Webserver started at http://127.21.83.62:34609/ using document root <none> and password file <none>
I20260807 07:02:22.823009 562 fs_manager.cc:359] Metadata directory not provided
I20260807 07:02:22.823107 562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:02:22.823419 562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:02:22.826237 562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "0935abe2905a450e8cbcfa9aead19321"
format_stamp: "Formatted at 2026-08-07 07:02:22 on dist-test-slave-ncp9"
server_key: "a30bf7ef96745e02a023f463d7bf5c4e"
server_key_iv: "d985268169adf345060dcc0438160e16"
server_key_version: "encryptionkey@0"
I20260807 07:02:22.826982 562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "0935abe2905a450e8cbcfa9aead19321"
format_stamp: "Formatted at 2026-08-07 07:02:22 on dist-test-slave-ncp9"
server_key: "a30bf7ef96745e02a023f463d7bf5c4e"
server_key_iv: "d985268169adf345060dcc0438160e16"
server_key_version: "encryptionkey@0"
I20260807 07:02:22.831740 562 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260807 07:02:22.835101 584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:22.836246 562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 07:02:22.836398 562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "0935abe2905a450e8cbcfa9aead19321"
format_stamp: "Formatted at 2026-08-07 07:02:22 on dist-test-slave-ncp9"
server_key: "a30bf7ef96745e02a023f463d7bf5c4e"
server_key_iv: "d985268169adf345060dcc0438160e16"
server_key_version: "encryptionkey@0"
I20260807 07:02:22.836589 562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:02:22.855511 562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:02:22.859279 562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:02:22.859536 562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:02:22.879495 562 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.62:35983
I20260807 07:02:22.879501 667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.62:35983 every 8 connection(s)
I20260807 07:02:22.881265 562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:02:22.882853 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 562
I20260807 07:02:22.883086 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:02:22.883766 21836 external_mini_cluster.cc:1428] Setting key 8921ddc5bc5e74288a09de49fd957664
I20260807 07:02:22.886648 668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 07:02:22 dist-test-slave-ncp9 krb5kdc[544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM
I20260807 07:02:22.900231 668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0935abe2905a450e8cbcfa9aead19321: Bootstrap starting.
I20260807 07:02:22.904610 668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0935abe2905a450e8cbcfa9aead19321: Neither blocks nor log segments found. Creating new log.
I20260807 07:02:22.906288 668 log.cc:824] T 00000000000000000000000000000000 P 0935abe2905a450e8cbcfa9aead19321: Log is configured to *not* fsync() on all Append() calls
I20260807 07:02:22.909420 668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0935abe2905a450e8cbcfa9aead19321: No bootstrap required, opened a new log
I20260807 07:02:22.909722 673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:22.885864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49756 (local address 127.21.83.62:35983)
0807 07:02:22.886766 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:22.886782 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:22.886835 (+ 53us) server_negotiation.cc:407] Connection header received
0807 07:02:22.889241 (+ 2406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:22.889276 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:22.889840 (+ 564us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:22.890337 (+ 497us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:22.891811 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:22.894002 (+ 2191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:22.895358 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:22.896428 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:22.896627 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:22.900629 (+ 4002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:22.900643 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:22.900664 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:22.900715 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:22.904794 (+ 4079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:22.905760 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:22.905862 (+ 102us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:22.905886 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:22.906160 (+ 274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:22.906691 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:22.906700 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:22.906703 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:22.907409 (+ 706us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:22.907701 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:22.908218 (+ 517us) server_negotiation.cc:299] Negotiation successful
0807 07:02:22.908658 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":386,"threads_started":1}
I20260807 07:02:22.917196 668 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0935abe2905a450e8cbcfa9aead19321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0935abe2905a450e8cbcfa9aead19321" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 35983 } }
I20260807 07:02:22.917533 668 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0935abe2905a450e8cbcfa9aead19321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:02:22.917654 668 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0935abe2905a450e8cbcfa9aead19321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0935abe2905a450e8cbcfa9aead19321, State: Initialized, Role: FOLLOWER
I20260807 07:02:22.918170 668 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0935abe2905a450e8cbcfa9aead19321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0935abe2905a450e8cbcfa9aead19321" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 35983 } }
I20260807 07:02:22.918388 668 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0935abe2905a450e8cbcfa9aead19321 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:02:22.918509 668 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0935abe2905a450e8cbcfa9aead19321 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:02:22.918669 668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0935abe2905a450e8cbcfa9aead19321 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:02:22.920756 668 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0935abe2905a450e8cbcfa9aead19321 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0935abe2905a450e8cbcfa9aead19321" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 35983 } }
I20260807 07:02:22.921247 668 leader_election.cc:302] T 00000000000000000000000000000000 P 0935abe2905a450e8cbcfa9aead19321 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0935abe2905a450e8cbcfa9aead19321; no voters:
I20260807 07:02:22.921818 668 leader_election.cc:288] T 00000000000000000000000000000000 P 0935abe2905a450e8cbcfa9aead19321 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 07:02:22.921937 675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0935abe2905a450e8cbcfa9aead19321 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:02:22.923092 675 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0935abe2905a450e8cbcfa9aead19321 [term 1 LEADER]: Becoming Leader. State: Replica: 0935abe2905a450e8cbcfa9aead19321, State: Running, Role: LEADER
I20260807 07:02:22.923568 675 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0935abe2905a450e8cbcfa9aead19321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0935abe2905a450e8cbcfa9aead19321" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 35983 } }
I20260807 07:02:22.923941 668 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0935abe2905a450e8cbcfa9aead19321 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:02:22.926359 678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0935abe2905a450e8cbcfa9aead19321 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0935abe2905a450e8cbcfa9aead19321. Latest consensus state: current_term: 1 leader_uuid: "0935abe2905a450e8cbcfa9aead19321" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0935abe2905a450e8cbcfa9aead19321" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 35983 } } }
I20260807 07:02:22.926704 678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0935abe2905a450e8cbcfa9aead19321 [sys.catalog]: This master's current role is: LEADER
I20260807 07:02:22.927142 676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0935abe2905a450e8cbcfa9aead19321 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0935abe2905a450e8cbcfa9aead19321" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0935abe2905a450e8cbcfa9aead19321" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 35983 } } }
I20260807 07:02:22.927440 676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0935abe2905a450e8cbcfa9aead19321 [sys.catalog]: This master's current role is: LEADER
I20260807 07:02:22.927558 688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:02:22.933002 688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:02:22.943645 688 catalog_manager.cc:1347] Generated new cluster ID: de54fcf39e644a42825c7c072999e6a8
I20260807 07:02:22.943862 688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:02:22.970310 688 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:02:22.971735 688 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:02:22.983342 688 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0935abe2905a450e8cbcfa9aead19321: Generated new TSK 0
I20260807 07:02:22.984392 688 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.21.83.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.83.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.83.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.1.keytab.
Entry for principal oryx/127.21.83.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.1.keytab.
I20260807 07:02:23.060806 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.1:0
--local_ip_for_outbound_sockets=127.21.83.1
--webserver_interface=127.21.83.1
--webserver_port=0
--tserver_master_addrs=127.21.83.62:35983
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.1.keytab
--principal=oryx/127.21.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:41903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:02:23.431770 704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:02:23.432061 704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:02:23.432189 704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:02:23.439540 704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:02:23.439658 704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:02:23.439824 704 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 07:02:23.439920 704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.1
I20260807 07:02:23.451421 704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:41903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.1.keytab
--principal=oryx/127.21.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.83.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.83.62:35983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.1
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:02:23.452601 704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:02:23.453979 704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:02:23.467768 714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:23.467818 716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:23.467990 713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:02:23.468772 704 server_base.cc:1034] running on GCE node
I20260807 07:02:23.469467 704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:02:23.470567 704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:02:23.471830 704 hybrid_clock.cc:648] HybridClock initialized: now 1786086143471735 us; error 57 us; skew 500 ppm
I20260807 07:02:23.472180 704 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:02:23 dist-test-slave-ncp9 krb5kdc[544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086143, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:02:23.475644 704 init.cc:377] Logged in from keytab as oryx/127.21.83.1@KRBTEST.COM (short username oryx)
I20260807 07:02:23.477322 704 webserver.cc:460] Webserver started at http://127.21.83.1:34355/ using document root <none> and password file <none>
I20260807 07:02:23.478080 704 fs_manager.cc:359] Metadata directory not provided
I20260807 07:02:23.478189 704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:02:23.478533 704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:02:23.482223 704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-0/data/instance:
uuid: "32317a2a19304a51a248edde90974bbb"
format_stamp: "Formatted at 2026-08-07 07:02:23 on dist-test-slave-ncp9"
server_key: "be75a41e0fee4daf743ae12f46ce6398"
server_key_iv: "f0ce7e383ddffd78588d965285b2cb25"
server_key_version: "encryptionkey@0"
I20260807 07:02:23.483165 704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-0/wal/instance:
uuid: "32317a2a19304a51a248edde90974bbb"
format_stamp: "Formatted at 2026-08-07 07:02:23 on dist-test-slave-ncp9"
server_key: "be75a41e0fee4daf743ae12f46ce6398"
server_key_iv: "f0ce7e383ddffd78588d965285b2cb25"
server_key_version: "encryptionkey@0"
I20260807 07:02:23.488771 704 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260807 07:02:23.492475 726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:23.493827 704 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 07:02:23.493991 704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-0/wal
uuid: "32317a2a19304a51a248edde90974bbb"
format_stamp: "Formatted at 2026-08-07 07:02:23 on dist-test-slave-ncp9"
server_key: "be75a41e0fee4daf743ae12f46ce6398"
server_key_iv: "f0ce7e383ddffd78588d965285b2cb25"
server_key_version: "encryptionkey@0"
I20260807 07:02:23.494221 704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:02:23.510820 704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:02:23.515138 704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:02:23.515374 704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:02:23.518253 704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:02:23.518368 704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:23.518537 704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:02:23.518597 704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 07:02:23 dist-test-slave-ncp9 krb5kdc[544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086143, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.83.1@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM
I20260807 07:02:23.541153 737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:23.518599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:42007 (local address 127.21.83.62:35983)
0807 07:02:23.519027 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:23.519033 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:23.520261 (+ 1228us) server_negotiation.cc:407] Connection header received
0807 07:02:23.522345 (+ 2084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:23.522352 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:23.522446 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:23.522543 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:23.524802 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:23.525931 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:23.528268 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:23.528960 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:23.529102 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:23.534408 (+ 5306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:23.534413 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:23.534417 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:23.534453 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:23.537424 (+ 2971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:23.538386 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:23.538393 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:23.538398 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:23.538575 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:23.539042 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:23.539047 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:23.539051 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:23.539429 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:23.539581 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:23.540374 (+ 793us) server_negotiation.cc:299] Negotiation successful
0807 07:02:23.540663 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":220,"threads_started":1}
I20260807 07:02:23.542045 739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:23.519288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:35983 (local address 127.21.83.1:42007)
0807 07:02:23.520095 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:23.520151 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:23.522042 (+ 1891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:23.522795 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:23.522807 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:23.523662 (+ 855us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:23.524523 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:23.524546 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:23.526135 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:23.526141 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:23.528076 (+ 1935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:23.528092 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:23.529230 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:23.529239 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:23.529519 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:23.530480 (+ 961us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:23.530517 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:23.534130 (+ 3613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:23.537613 (+ 3483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:23.537623 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:23.537643 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:23.538251 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:23.538727 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:23.538730 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:23.538733 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:23.538947 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:23.539544 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:23.539550 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:23.539973 (+ 423us) client_negotiation.cc:769] Sending connection context
0807 07:02:23.540244 (+ 271us) client_negotiation.cc:240] Negotiation successful
0807 07:02:23.540699 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":363,"threads_started":1}
I20260807 07:02:23.566236 704 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.1:44117
I20260807 07:02:23.566262 904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.1:44117 every 8 connection(s)
I20260807 07:02:23.567718 704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
I20260807 07:02:23.568118 737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:23.552677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:49549 (local address 127.21.83.62:35983)
0807 07:02:23.552934 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:23.552941 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:23.553614 (+ 673us) server_negotiation.cc:407] Connection header received
0807 07:02:23.553795 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:23.553802 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:23.553887 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:23.553990 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:23.555518 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:23.556585 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:23.558617 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:23.559260 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:23.559389 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:23.562001 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:23.562009 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:23.562013 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:23.562052 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:23.565350 (+ 3298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:23.566296 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:23.566303 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:23.566307 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:23.566464 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:23.566833 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:23.566836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:23.566838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:23.567101 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:23.567221 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:23.567689 (+ 468us) server_negotiation.cc:299] Negotiation successful
0807 07:02:23.567878 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260807 07:02:23.568377 841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:23.552704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:35983 (local address 127.21.83.1:49549)
0807 07:02:23.553510 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:23.553525 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:23.553670 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:23.554057 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:23.554063 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:23.554794 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:23.555317 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:23.555334 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:23.556761 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:23.556770 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:23.558457 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:23.558475 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:23.559480 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:23.559488 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:23.559679 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:23.560405 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:23.560431 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:23.561712 (+ 1281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:23.565518 (+ 3806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:23.565523 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:23.565527 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:23.566149 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:23.566576 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:23.566581 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:23.566584 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:23.566735 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:23.567227 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:23.567232 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:23.567453 (+ 221us) client_negotiation.cc:769] Sending connection context
0807 07:02:23.567604 (+ 151us) client_negotiation.cc:240] Negotiation successful
0807 07:02:23.567919 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":544,"threads_started":1}
I20260807 07:02:23.570024 905 heartbeater.cc:347] Connected to a master server at 127.21.83.62:35983
I20260807 07:02:23.570267 905 heartbeater.cc:464] Registering TS with master...
I20260807 07:02:23.571043 905 heartbeater.cc:511] Master 127.21.83.62:35983 requested a full tablet report, sending...
I20260807 07:02:23.572414 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 704
I20260807 07:02:23.572659 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-0/wal/instance
I20260807 07:02:23.573426 603 ts_manager.cc:194] Registered new tserver with Master: 32317a2a19304a51a248edde90974bbb (127.21.83.1:44117)
I20260807 07:02:23.573529 21836 external_mini_cluster.cc:1428] Setting key 945f8e3425c467855e10cb056ce449b2
I20260807 07:02:23.575066 603 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.83.1@KRBTEST.COM'} at 127.21.83.1:42007
WARNING: no policy specified for oryx/127.21.83.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.83.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.83.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.2.keytab.
Entry for principal oryx/127.21.83.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.2.keytab.
I20260807 07:02:23.629981 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.2:0
--local_ip_for_outbound_sockets=127.21.83.2
--webserver_interface=127.21.83.2
--webserver_port=0
--tserver_master_addrs=127.21.83.62:35983
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.2.keytab
--principal=oryx/127.21.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:41903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:02:23.974741 912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:02:23.975068 912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:02:23.975222 912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:02:23.983474 912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:02:23.983958 912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:02:23.984176 912 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 07:02:23.984314 912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.2
I20260807 07:02:23.995072 912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:41903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.2.keytab
--principal=oryx/127.21.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.83.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.83.62:35983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.2
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:02:23.996551 912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:02:23.998035 912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:02:24.013372 921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:24.013458 920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:24.013411 923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:02:24.014400 912 server_base.cc:1034] running on GCE node
I20260807 07:02:24.015092 912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:02:24.016106 912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:02:24.017328 912 hybrid_clock.cc:648] HybridClock initialized: now 1786086144017258 us; error 44 us; skew 500 ppm
I20260807 07:02:24.017638 912 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:02:24 dist-test-slave-ncp9 krb5kdc[544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086144, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:02:24.024020 912 init.cc:377] Logged in from keytab as oryx/127.21.83.2@KRBTEST.COM (short username oryx)
I20260807 07:02:24.025761 912 webserver.cc:460] Webserver started at http://127.21.83.2:37725/ using document root <none> and password file <none>
I20260807 07:02:24.026508 912 fs_manager.cc:359] Metadata directory not provided
I20260807 07:02:24.026625 912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:02:24.026957 912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:02:24.030211 912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-1/data/instance:
uuid: "a4336684d7414e61b17d557ddde64653"
format_stamp: "Formatted at 2026-08-07 07:02:24 on dist-test-slave-ncp9"
server_key: "a44fe3b0fd3f949a77f5a5e1206bab14"
server_key_iv: "b5df9b2b092e562d1d12f3dc35a1318e"
server_key_version: "encryptionkey@0"
I20260807 07:02:24.031080 912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-1/wal/instance:
uuid: "a4336684d7414e61b17d557ddde64653"
format_stamp: "Formatted at 2026-08-07 07:02:24 on dist-test-slave-ncp9"
server_key: "a44fe3b0fd3f949a77f5a5e1206bab14"
server_key_iv: "b5df9b2b092e562d1d12f3dc35a1318e"
server_key_version: "encryptionkey@0"
I20260807 07:02:24.036593 912 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260807 07:02:24.040491 932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:24.042007 912 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 07:02:24.042168 912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-1/wal
uuid: "a4336684d7414e61b17d557ddde64653"
format_stamp: "Formatted at 2026-08-07 07:02:24 on dist-test-slave-ncp9"
server_key: "a44fe3b0fd3f949a77f5a5e1206bab14"
server_key_iv: "b5df9b2b092e562d1d12f3dc35a1318e"
server_key_version: "encryptionkey@0"
I20260807 07:02:24.042418 912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:02:24.060577 912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:02:24.064983 912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:02:24.065222 912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:02:24.067915 912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:02:24.068019 912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:24.068223 912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:02:24.068290 912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 07:02:24 dist-test-slave-ncp9 krb5kdc[544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086144, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.83.2@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM
I20260807 07:02:24.095081 947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:24.068981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:35983 (local address 127.21.83.2:59169)
0807 07:02:24.069893 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:24.069942 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:24.072548 (+ 2606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:24.073422 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:24.073438 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:24.074329 (+ 891us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:24.075303 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:24.075337 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:24.076663 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:24.076669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:24.079177 (+ 2508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:24.079198 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:24.080211 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:24.080218 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:24.080579 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:24.081517 (+ 938us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:24.081546 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:24.086008 (+ 4462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:24.089763 (+ 3755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:24.089774 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:24.089794 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:24.090409 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:24.090973 (+ 564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:24.090979 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:24.090983 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:24.091289 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:24.091973 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:24.091983 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:24.092524 (+ 541us) client_negotiation.cc:769] Sending connection context
0807 07:02:24.092845 (+ 321us) client_negotiation.cc:240] Negotiation successful
0807 07:02:24.093474 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":377,"threads_started":1}
I20260807 07:02:24.096275 737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:24.068360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:59169 (local address 127.21.83.62:35983)
0807 07:02:24.068572 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:24.068580 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:24.070083 (+ 1503us) server_negotiation.cc:407] Connection header received
0807 07:02:24.072882 (+ 2799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:24.072910 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:24.073015 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:24.073131 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:24.075526 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:24.076461 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:24.079361 (+ 2900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:24.079914 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:24.080047 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:24.086283 (+ 6236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:24.086289 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:24.086292 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:24.086331 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:24.089601 (+ 3270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:24.090632 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:24.090640 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:24.090644 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:24.090837 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:24.091463 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:24.091469 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:24.091473 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:24.091796 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:24.095583 (+ 3787us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:24.095745 (+ 162us) server_negotiation.cc:299] Negotiation successful
0807 07:02:24.096013 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 07:02:24.116526 737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:24.101756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:38669 (local address 127.21.83.62:35983)
0807 07:02:24.101911 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:24.101915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:24.102507 (+ 592us) server_negotiation.cc:407] Connection header received
0807 07:02:24.102687 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:24.102694 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:24.102774 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:24.102873 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:24.104494 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:24.105652 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:24.107396 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:24.107825 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:24.107933 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:24.110207 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:24.110215 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:24.110220 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:24.110254 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:24.113020 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:24.114242 (+ 1222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:24.114249 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:24.114253 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:24.114423 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:24.114837 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:24.114842 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:24.114845 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:24.115309 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:24.115464 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:24.115940 (+ 476us) server_negotiation.cc:299] Negotiation successful
0807 07:02:24.116166 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260807 07:02:24.116637 912 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.2:40429
I20260807 07:02:24.116768 1059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:24.101784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:35983 (local address 127.21.83.2:38669)
0807 07:02:24.102402 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:24.102418 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:24.102566 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:24.102943 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:24.102949 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:24.103716 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:24.104264 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:24.104286 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:24.105781 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:24.105785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:24.107260 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:24.107274 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:24.108008 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:24.108015 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:24.108165 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:24.108816 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:24.108833 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:24.109964 (+ 1131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:24.113152 (+ 3188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:24.113157 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:24.113162 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:24.114050 (+ 888us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:24.114555 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:24.114560 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:24.114564 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:24.114731 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:24.115451 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:24.115459 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:24.115670 (+ 211us) client_negotiation.cc:769] Sending connection context
0807 07:02:24.115836 (+ 166us) client_negotiation.cc:240] Negotiation successful
0807 07:02:24.116272 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":326,"threads_started":1}
I20260807 07:02:24.117216 1109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.2:40429 every 8 connection(s)
I20260807 07:02:24.118191 912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
I20260807 07:02:24.120663 1110 heartbeater.cc:347] Connected to a master server at 127.21.83.62:35983
I20260807 07:02:24.120883 1110 heartbeater.cc:464] Registering TS with master...
I20260807 07:02:24.121605 1110 heartbeater.cc:511] Master 127.21.83.62:35983 requested a full tablet report, sending...
I20260807 07:02:24.122785 603 ts_manager.cc:194] Registered new tserver with Master: a4336684d7414e61b17d557ddde64653 (127.21.83.2:40429)
I20260807 07:02:24.123553 603 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.83.2@KRBTEST.COM'} at 127.21.83.2:59169
I20260807 07:02:24.127164 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 912
I20260807 07:02:24.127326 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-1/wal/instance
I20260807 07:02:24.127733 21836 external_mini_cluster.cc:1428] Setting key 8e65c99ad715beb05ddf8fcb0a41813e
WARNING: no policy specified for oryx/127.21.83.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.83.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.83.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.3.keytab.
Entry for principal oryx/127.21.83.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.3.keytab.
I20260807 07:02:24.187983 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.3:0
--local_ip_for_outbound_sockets=127.21.83.3
--webserver_interface=127.21.83.3
--webserver_port=0
--tserver_master_addrs=127.21.83.62:35983
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.3.keytab
--principal=oryx/127.21.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:41903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:02:24.528417 1115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:02:24.528724 1115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:02:24.528872 1115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:02:24.536689 1115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:02:24.536821 1115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:02:24.537003 1115 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 07:02:24.537135 1115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.3
I20260807 07:02:24.547835 1115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:41903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.3.keytab
--principal=oryx/127.21.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.83.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.83.62:35983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.3
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:02:24.549232 1115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:02:24.550580 1115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:02:24.563865 1123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:24.563829 1124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:24.563910 1128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:02:24.565461 1115 server_base.cc:1034] running on GCE node
I20260807 07:02:24.566352 1115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:02:24.567698 1115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:02:24.569046 1115 hybrid_clock.cc:648] HybridClock initialized: now 1786086144568955 us; error 49 us; skew 500 ppm
I20260807 07:02:24.569442 1115 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:02:24 dist-test-slave-ncp9 krb5kdc[544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086144, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:02:24.573411 1115 init.cc:377] Logged in from keytab as oryx/127.21.83.3@KRBTEST.COM (short username oryx)
I20260807 07:02:24.575013 1115 webserver.cc:460] Webserver started at http://127.21.83.3:40793/ using document root <none> and password file <none>
I20260807 07:02:24.575753 1115 fs_manager.cc:359] Metadata directory not provided
I20260807 07:02:24.575846 1115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:02:24.576135 1115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:02:24.578150 905 heartbeater.cc:503] Master 127.21.83.62:35983 was elected leader, sending a full tablet report...
I20260807 07:02:24.579448 1115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-2/data/instance:
uuid: "d02e909ab20f4a12b000cb5ca64212e7"
format_stamp: "Formatted at 2026-08-07 07:02:24 on dist-test-slave-ncp9"
server_key: "aaa0f70740ca20b0637a5d60fffecfdc"
server_key_iv: "6290497e2b1e2a04c29d54239bdd6a1c"
server_key_version: "encryptionkey@0"
I20260807 07:02:24.580303 1115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-2/wal/instance:
uuid: "d02e909ab20f4a12b000cb5ca64212e7"
format_stamp: "Formatted at 2026-08-07 07:02:24 on dist-test-slave-ncp9"
server_key: "aaa0f70740ca20b0637a5d60fffecfdc"
server_key_iv: "6290497e2b1e2a04c29d54239bdd6a1c"
server_key_version: "encryptionkey@0"
I20260807 07:02:24.585412 1115 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260807 07:02:24.588672 1136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:24.589936 1115 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 07:02:24.590088 1115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-2/wal
uuid: "d02e909ab20f4a12b000cb5ca64212e7"
format_stamp: "Formatted at 2026-08-07 07:02:24 on dist-test-slave-ncp9"
server_key: "aaa0f70740ca20b0637a5d60fffecfdc"
server_key_iv: "6290497e2b1e2a04c29d54239bdd6a1c"
server_key_version: "encryptionkey@0"
I20260807 07:02:24.590279 1115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:02:24.624425 1115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:02:24.628492 1115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:02:24.628762 1115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:02:24.631521 1115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:02:24.631604 1115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:24.631757 1115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:02:24.631815 1115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 07:02:24 dist-test-slave-ncp9 krb5kdc[544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086144, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.83.3@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM
I20260807 07:02:24.657538 1151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:24.632305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:51157 (local address 127.21.83.62:35983)
0807 07:02:24.632770 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:24.632776 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:24.634039 (+ 1263us) server_negotiation.cc:407] Connection header received
0807 07:02:24.636939 (+ 2900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:24.636948 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:24.637053 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:24.637141 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:24.640081 (+ 2940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:24.641340 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:24.644192 (+ 2852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:24.644812 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:24.644983 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:24.649628 (+ 4645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:24.649636 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:24.649640 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:24.649683 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:24.652759 (+ 3076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:24.653974 (+ 1215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:24.653980 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:24.653983 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:24.654150 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:24.654842 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:24.654848 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:24.654852 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:24.655291 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:24.655468 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:24.656753 (+ 1285us) server_negotiation.cc:299] Negotiation successful
0807 07:02:24.657071 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":244,"threads_started":1}
I20260807 07:02:24.658717 1152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:24.632971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:35983 (local address 127.21.83.3:51157)
0807 07:02:24.633846 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:24.633908 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:24.636511 (+ 2603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:24.637755 (+ 1244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:24.637767 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:24.638690 (+ 923us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:24.639791 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:24.639823 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:24.641588 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:24.641593 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:24.644025 (+ 2432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:24.644046 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:24.645311 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:24.645318 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:24.645624 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:24.646541 (+ 917us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:24.646568 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:24.649389 (+ 2821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:24.652980 (+ 3591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:24.652991 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:24.653015 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:24.653828 (+ 813us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:24.654291 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:24.654297 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:24.654301 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:24.654681 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:24.655464 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:24.655474 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:24.656090 (+ 616us) client_negotiation.cc:769] Sending connection context
0807 07:02:24.656464 (+ 374us) client_negotiation.cc:240] Negotiation successful
0807 07:02:24.657174 (+ 710us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":339,"threads_started":1}
I20260807 07:02:24.674666 1115 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.3:35127
I20260807 07:02:24.674748 1322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.3:35127 every 8 connection(s)
I20260807 07:02:24.675982 1115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
I20260807 07:02:24.677241 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 1115
I20260807 07:02:24.677459 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-2/wal/instance
I20260807 07:02:24.679127 21836 external_mini_cluster.cc:1428] Setting key 808add2d6ae00a9a4950774ad5d4e5f6
I20260807 07:02:24.678268 1323 heartbeater.cc:347] Connected to a master server at 127.21.83.62:35983
I20260807 07:02:24.680099 1323 heartbeater.cc:464] Registering TS with master...
I20260807 07:02:24.680765 1295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:24.666155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:35983 (local address 127.21.83.3:49597)
0807 07:02:24.666991 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:24.667015 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:24.667184 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:24.667681 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:24.667690 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:24.668302 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:24.668760 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:24.668778 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:24.670253 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:24.670257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:24.671273 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:24.671285 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:24.672211 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:24.672215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:24.672328 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:24.672981 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:24.672998 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:24.673927 (+ 929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:24.677186 (+ 3259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:24.677192 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:24.677196 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:24.677741 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:24.678728 (+ 987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:24.678733 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:24.678735 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:24.678888 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:24.679567 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:24.679572 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:24.679792 (+ 220us) client_negotiation.cc:769] Sending connection context
0807 07:02:24.679952 (+ 160us) client_negotiation.cc:240] Negotiation successful
0807 07:02:24.680297 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":515,"threads_started":1}
I20260807 07:02:24.681052 1323 heartbeater.cc:511] Master 127.21.83.62:35983 requested a full tablet report, sending...
I20260807 07:02:24.681185 1151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:24.666059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:49597 (local address 127.21.83.62:35983)
0807 07:02:24.666252 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:24.666258 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:24.667104 (+ 846us) server_negotiation.cc:407] Connection header received
0807 07:02:24.667400 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:24.667406 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:24.667513 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:24.667645 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:24.668976 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:24.670122 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:24.671397 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:24.672024 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:24.672165 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:24.674119 (+ 1954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:24.674124 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:24.674127 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:24.674168 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:24.677025 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:24.678428 (+ 1403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:24.678434 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:24.678437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:24.678594 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:24.679033 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:24.679038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:24.679042 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:24.679411 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:24.679559 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:24.680586 (+ 1027us) server_negotiation.cc:299] Negotiation successful
0807 07:02:24.680840 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260807 07:02:24.682466 603 ts_manager.cc:194] Registered new tserver with Master: d02e909ab20f4a12b000cb5ca64212e7 (127.21.83.3:35127)
I20260807 07:02:24.683300 603 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.83.3@KRBTEST.COM'} at 127.21.83.3:51157
I20260807 07:02:24.684341 21836 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 07:02:24 dist-test-slave-ncp9 krb5kdc[544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086144, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 07:02:24 dist-test-slave-ncp9 krb5kdc[544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:24 dist-test-slave-ncp9 krb5kdc[544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
W20260807 07:02:24.743539 1151 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:24.733415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49758 (local address 127.21.83.62:35983)
0807 07:02:24.733637 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:24.733643 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:24.733884 (+ 241us) server_negotiation.cc:407] Connection header received
0807 07:02:24.734078 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:24.734083 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:24.734163 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:24.734247 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:24.735612 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:24.736343 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:24.737667 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:24.738067 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:24.738177 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:24.743322 (+ 5145us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49758: BlockingRecv error: recv got EOF from 127.0.0.1:49758 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 07 07:02:24 dist-test-slave-ncp9 krb5kdc[544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086144, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM
I20260807 07:02:24.764726 1151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:24.747038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49768 (local address 127.21.83.62:35983)
0807 07:02:24.747208 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:24.747212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:24.747521 (+ 309us) server_negotiation.cc:407] Connection header received
0807 07:02:24.747668 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:24.747672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:24.747735 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:24.747819 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:24.748990 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:24.749727 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:24.750963 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:24.751393 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:24.751498 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:24.759321 (+ 7823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:24.759328 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:24.759332 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:24.759375 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:24.762067 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:24.762825 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:24.762830 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:24.762832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:24.762975 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:24.763340 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:24.763345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:24.763347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:24.763627 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:24.763896 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:24.764195 (+ 299us) server_negotiation.cc:299] Negotiation successful
0807 07:02:24.764429 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260807 07:02:24.768584 604 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49768:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 07:02:24.775425 604 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 07:02:24.796030 1344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:24.786828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.2:40429 (local address 127.0.0.1:41708)
0807 07:02:24.787735 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:24.787770 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:24.787953 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:24.789108 (+ 1155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:24.789115 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:24.789168 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:02:24.789568 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:24.789583 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:24.791479 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:24.791484 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:24.793124 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:24.793139 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:24.794972 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:24.794979 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:24.795259 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:24.795321 (+ 62us) client_negotiation.cc:769] Sending connection context
0807 07:02:24.795453 (+ 132us) client_negotiation.cc:240] Negotiation successful
0807 07:02:24.795637 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"mutex_wait_us":41,"thread_start_us":526,"threads_started":1}
I20260807 07:02:24.796447 1348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:24.786985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41708 (local address 127.21.83.2:40429)
0807 07:02:24.788451 (+ 1466us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:24.788459 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:24.788486 (+ 27us) server_negotiation.cc:407] Connection header received
0807 07:02:24.788586 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:24.788593 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:24.788836 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:24.789791 (+ 955us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:02:24.789992 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:24.791355 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:24.793335 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:24.794748 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:24.794867 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:24.795065 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:24.795715 (+ 650us) server_negotiation.cc:299] Negotiation successful
0807 07:02:24.795886 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1323,"thread_start_us":1207,"threads_started":1}
I20260807 07:02:24.798115 1349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:24.787400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:44117 (local address 127.0.0.1:58544)
0807 07:02:24.788486 (+ 1086us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:24.788504 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:24.788614 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:24.790292 (+ 1678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:24.790297 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:24.790321 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:02:24.790761 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:24.790773 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:24.792756 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:24.792762 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:24.795343 (+ 2581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:24.795356 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:24.797151 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:24.797158 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:24.797466 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:24.797498 (+ 32us) client_negotiation.cc:769] Sending connection context
0807 07:02:24.797582 (+ 84us) client_negotiation.cc:240] Negotiation successful
0807 07:02:24.797725 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":942,"mutex_wait_us":576,"thread_start_us":969,"threads_started":1}
I20260807 07:02:24.798513 1350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:24.788110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.3:35127 (local address 127.0.0.1:41542)
0807 07:02:24.789085 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:24.789102 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:24.789246 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:24.790749 (+ 1503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:24.790754 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:24.790777 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:02:24.791193 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:24.791206 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:24.793249 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:24.793254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:24.795494 (+ 2240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:24.795508 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:24.797268 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:24.797273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:24.797567 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:24.797602 (+ 35us) client_negotiation.cc:769] Sending connection context
0807 07:02:24.797758 (+ 156us) client_negotiation.cc:240] Negotiation successful
0807 07:02:24.797893 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":861,"mutex_wait_us":98,"spinlock_wait_cycles":1792,"thread_start_us":714,"threads_started":1}
I20260807 07:02:24.798671 1353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:24.789075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41542 (local address 127.21.83.3:35127)
0807 07:02:24.790186 (+ 1111us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:24.790193 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:24.790223 (+ 30us) server_negotiation.cc:407] Connection header received
0807 07:02:24.790309 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:24.790315 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:24.790516 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:24.791392 (+ 876us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:02:24.791699 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:24.793126 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:24.795669 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:24.797029 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:24.797413 (+ 384us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:24.797642 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:24.797715 (+ 73us) server_negotiation.cc:299] Negotiation successful
0807 07:02:24.797861 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":936,"thread_start_us":744,"threads_started":1}
I20260807 07:02:24.799476 1352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:24.788526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58544 (local address 127.21.83.1:44117)
0807 07:02:24.789729 (+ 1203us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:24.789737 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:24.789765 (+ 28us) server_negotiation.cc:407] Connection header received
0807 07:02:24.789858 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:24.789865 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:24.790071 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:24.790973 (+ 902us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:02:24.791194 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:24.792614 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:24.795504 (+ 2890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:24.796948 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:24.797082 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:24.797334 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:24.799008 (+ 1674us) server_negotiation.cc:299] Negotiation successful
0807 07:02:24.799192 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1060,"thread_start_us":924,"threads_started":1}
I20260807 07:02:24.800447 1005 tablet_service.cc:1467] Processing CreateTablet for tablet a03e3b06eee645618f04243666116c36 (DEFAULT_TABLE table=test-table [id=4c3cf324684c4740807571abb332e107]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:02:24.801999 1005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a03e3b06eee645618f04243666116c36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:02:24.802163 799 tablet_service.cc:1467] Processing CreateTablet for tablet a03e3b06eee645618f04243666116c36 (DEFAULT_TABLE table=test-table [id=4c3cf324684c4740807571abb332e107]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:02:24.802206 1217 tablet_service.cc:1467] Processing CreateTablet for tablet a03e3b06eee645618f04243666116c36 (DEFAULT_TABLE table=test-table [id=4c3cf324684c4740807571abb332e107]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:02:24.803400 799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a03e3b06eee645618f04243666116c36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:02:24.803608 1217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a03e3b06eee645618f04243666116c36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:02:24.815097 1354 tablet_bootstrap.cc:492] T a03e3b06eee645618f04243666116c36 P 32317a2a19304a51a248edde90974bbb: Bootstrap starting.
I20260807 07:02:24.817404 1360 tablet_bootstrap.cc:492] T a03e3b06eee645618f04243666116c36 P d02e909ab20f4a12b000cb5ca64212e7: Bootstrap starting.
I20260807 07:02:24.818856 1356 tablet_bootstrap.cc:492] T a03e3b06eee645618f04243666116c36 P a4336684d7414e61b17d557ddde64653: Bootstrap starting.
I20260807 07:02:24.820382 1354 tablet_bootstrap.cc:654] T a03e3b06eee645618f04243666116c36 P 32317a2a19304a51a248edde90974bbb: Neither blocks nor log segments found. Creating new log.
I20260807 07:02:24.820690 1360 tablet_bootstrap.cc:654] T a03e3b06eee645618f04243666116c36 P d02e909ab20f4a12b000cb5ca64212e7: Neither blocks nor log segments found. Creating new log.
I20260807 07:02:24.821694 1360 log.cc:824] T a03e3b06eee645618f04243666116c36 P d02e909ab20f4a12b000cb5ca64212e7: Log is configured to *not* fsync() on all Append() calls
I20260807 07:02:24.821763 1354 log.cc:824] T a03e3b06eee645618f04243666116c36 P 32317a2a19304a51a248edde90974bbb: Log is configured to *not* fsync() on all Append() calls
I20260807 07:02:24.822073 1356 tablet_bootstrap.cc:654] T a03e3b06eee645618f04243666116c36 P a4336684d7414e61b17d557ddde64653: Neither blocks nor log segments found. Creating new log.
I20260807 07:02:24.823275 1356 log.cc:824] T a03e3b06eee645618f04243666116c36 P a4336684d7414e61b17d557ddde64653: Log is configured to *not* fsync() on all Append() calls
I20260807 07:02:24.824704 1360 tablet_bootstrap.cc:492] T a03e3b06eee645618f04243666116c36 P d02e909ab20f4a12b000cb5ca64212e7: No bootstrap required, opened a new log
I20260807 07:02:24.825019 1360 ts_tablet_manager.cc:1397] T a03e3b06eee645618f04243666116c36 P d02e909ab20f4a12b000cb5ca64212e7: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.002s
I20260807 07:02:24.825413 1354 tablet_bootstrap.cc:492] T a03e3b06eee645618f04243666116c36 P 32317a2a19304a51a248edde90974bbb: No bootstrap required, opened a new log
I20260807 07:02:24.825664 1354 ts_tablet_manager.cc:1397] T a03e3b06eee645618f04243666116c36 P 32317a2a19304a51a248edde90974bbb: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260807 07:02:24.826599 1356 tablet_bootstrap.cc:492] T a03e3b06eee645618f04243666116c36 P a4336684d7414e61b17d557ddde64653: No bootstrap required, opened a new log
I20260807 07:02:24.826858 1356 ts_tablet_manager.cc:1397] T a03e3b06eee645618f04243666116c36 P a4336684d7414e61b17d557ddde64653: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260807 07:02:24.832597 1360 raft_consensus.cc:348] T a03e3b06eee645618f04243666116c36 P d02e909ab20f4a12b000cb5ca64212e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } } peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } }
I20260807 07:02:24.832958 1360 raft_consensus.cc:374] T a03e3b06eee645618f04243666116c36 P d02e909ab20f4a12b000cb5ca64212e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:02:24.833066 1360 raft_consensus.cc:729] T a03e3b06eee645618f04243666116c36 P d02e909ab20f4a12b000cb5ca64212e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d02e909ab20f4a12b000cb5ca64212e7, State: Initialized, Role: FOLLOWER
I20260807 07:02:24.833606 1360 consensus_queue.cc:260] T a03e3b06eee645618f04243666116c36 P d02e909ab20f4a12b000cb5ca64212e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } } peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } }
I20260807 07:02:24.834635 1323 heartbeater.cc:503] Master 127.21.83.62:35983 was elected leader, sending a full tablet report...
I20260807 07:02:24.835297 1360 ts_tablet_manager.cc:1428] T a03e3b06eee645618f04243666116c36 P d02e909ab20f4a12b000cb5ca64212e7: Time spent starting tablet: real 0.010s user 0.009s sys 0.001s
I20260807 07:02:24.835886 1354 raft_consensus.cc:348] T a03e3b06eee645618f04243666116c36 P 32317a2a19304a51a248edde90974bbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } } peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } }
I20260807 07:02:24.836294 1354 raft_consensus.cc:374] T a03e3b06eee645618f04243666116c36 P 32317a2a19304a51a248edde90974bbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:02:24.836388 1354 raft_consensus.cc:729] T a03e3b06eee645618f04243666116c36 P 32317a2a19304a51a248edde90974bbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32317a2a19304a51a248edde90974bbb, State: Initialized, Role: FOLLOWER
I20260807 07:02:24.836987 1354 consensus_queue.cc:260] T a03e3b06eee645618f04243666116c36 P 32317a2a19304a51a248edde90974bbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } } peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } }
I20260807 07:02:24.837000 1356 raft_consensus.cc:348] T a03e3b06eee645618f04243666116c36 P a4336684d7414e61b17d557ddde64653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } } peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } }
I20260807 07:02:24.837394 1356 raft_consensus.cc:374] T a03e3b06eee645618f04243666116c36 P a4336684d7414e61b17d557ddde64653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:02:24.837503 1356 raft_consensus.cc:729] T a03e3b06eee645618f04243666116c36 P a4336684d7414e61b17d557ddde64653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4336684d7414e61b17d557ddde64653, State: Initialized, Role: FOLLOWER
I20260807 07:02:24.838130 1356 consensus_queue.cc:260] T a03e3b06eee645618f04243666116c36 P a4336684d7414e61b17d557ddde64653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } } peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } }
I20260807 07:02:24.839156 1110 heartbeater.cc:503] Master 127.21.83.62:35983 was elected leader, sending a full tablet report...
I20260807 07:02:24.839722 1356 ts_tablet_manager.cc:1428] T a03e3b06eee645618f04243666116c36 P a4336684d7414e61b17d557ddde64653: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260807 07:02:24.840544 1354 ts_tablet_manager.cc:1428] T a03e3b06eee645618f04243666116c36 P 32317a2a19304a51a248edde90974bbb: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
W20260807 07:02:24.870285 1111 tablet.cc:2369] T a03e3b06eee645618f04243666116c36 P a4336684d7414e61b17d557ddde64653: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 07:02:24.928864 1324 tablet.cc:2369] T a03e3b06eee645618f04243666116c36 P d02e909ab20f4a12b000cb5ca64212e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 07:02:25.070626 906 tablet.cc:2369] T a03e3b06eee645618f04243666116c36 P 32317a2a19304a51a248edde90974bbb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 07:02:25.081923 1366 raft_consensus.cc:482] T a03e3b06eee645618f04243666116c36 P a4336684d7414e61b17d557ddde64653 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:02:25.082227 1366 raft_consensus.cc:504] T a03e3b06eee645618f04243666116c36 P a4336684d7414e61b17d557ddde64653 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } } peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } }
I20260807 07:02:25.083902 1366 leader_election.cc:288] T a03e3b06eee645618f04243666116c36 P a4336684d7414e61b17d557ddde64653 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d02e909ab20f4a12b000cb5ca64212e7 (127.21.83.3:35127), 32317a2a19304a51a248edde90974bbb (127.21.83.1:44117)
I20260807 07:02:25.091254 1353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:25.084363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:35291 (local address 127.21.83.3:35127)
0807 07:02:25.084566 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:25.084572 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:25.086009 (+ 1437us) server_negotiation.cc:407] Connection header received
0807 07:02:25.086171 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:25.086177 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:25.086257 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:25.086355 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:02:25.086920 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:25.087647 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:25.089592 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:25.090340 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:25.090436 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:25.090535 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:25.090880 (+ 345us) server_negotiation.cc:299] Negotiation successful
0807 07:02:25.091012 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 07:02:25.091257 1352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:25.084739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:37093 (local address 127.21.83.1:44117)
0807 07:02:25.084971 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:25.084978 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:25.085695 (+ 717us) server_negotiation.cc:407] Connection header received
0807 07:02:25.085921 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:25.085928 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:25.086021 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:25.086112 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:02:25.086745 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:25.087483 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:25.089611 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:25.090399 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:25.090533 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:25.090631 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:25.090901 (+ 270us) server_negotiation.cc:299] Negotiation successful
0807 07:02:25.091020 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 07:02:25.091389 1367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:25.084371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.3:35127 (local address 127.21.83.2:35291)
0807 07:02:25.085886 (+ 1515us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:25.085908 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:25.086035 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:25.086381 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:25.086384 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:25.086405 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:02:25.086760 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:25.086772 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:25.087785 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:25.087789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:25.089444 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:25.089457 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:25.090559 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:25.090566 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:25.090727 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:25.090752 (+ 25us) client_negotiation.cc:769] Sending connection context
0807 07:02:25.090822 (+ 70us) client_negotiation.cc:240] Negotiation successful
0807 07:02:25.090936 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1387,"mutex_wait_us":34,"thread_start_us":1194,"threads_started":1}
I20260807 07:02:25.091444 1368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:25.084349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:44117 (local address 127.21.83.2:37093)
0807 07:02:25.085557 (+ 1208us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:25.085578 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:25.085757 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:25.086145 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:25.086150 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:25.086187 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:02:25.086582 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:25.086594 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:25.087618 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:25.087623 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:25.089460 (+ 1837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:25.089474 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:25.090629 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:25.090634 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:25.090751 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:25.090774 (+ 23us) client_negotiation.cc:769] Sending connection context
0807 07:02:25.090860 (+ 86us) client_negotiation.cc:240] Negotiation successful
0807 07:02:25.090979 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1055,"mutex_wait_us":35,"spinlock_wait_cycles":6400,"thread_start_us":819,"threads_started":1}
I20260807 07:02:25.092314 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03e3b06eee645618f04243666116c36" candidate_uuid: "a4336684d7414e61b17d557ddde64653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d02e909ab20f4a12b000cb5ca64212e7" is_pre_election: true
I20260807 07:02:25.092312 828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03e3b06eee645618f04243666116c36" candidate_uuid: "a4336684d7414e61b17d557ddde64653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32317a2a19304a51a248edde90974bbb" is_pre_election: true
I20260807 07:02:25.092762 828 raft_consensus.cc:2454] T a03e3b06eee645618f04243666116c36 P 32317a2a19304a51a248edde90974bbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4336684d7414e61b17d557ddde64653 in term 0.
I20260807 07:02:25.092756 1241 raft_consensus.cc:2454] T a03e3b06eee645618f04243666116c36 P d02e909ab20f4a12b000cb5ca64212e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4336684d7414e61b17d557ddde64653 in term 0.
I20260807 07:02:25.093364 934 leader_election.cc:302] T a03e3b06eee645618f04243666116c36 P a4336684d7414e61b17d557ddde64653 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4336684d7414e61b17d557ddde64653, d02e909ab20f4a12b000cb5ca64212e7; no voters:
I20260807 07:02:25.093760 1366 raft_consensus.cc:2781] T a03e3b06eee645618f04243666116c36 P a4336684d7414e61b17d557ddde64653 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 07:02:25.093910 1366 raft_consensus.cc:482] T a03e3b06eee645618f04243666116c36 P a4336684d7414e61b17d557ddde64653 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:02:25.094010 1366 raft_consensus.cc:3037] T a03e3b06eee645618f04243666116c36 P a4336684d7414e61b17d557ddde64653 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:02:25.096189 1366 raft_consensus.cc:504] T a03e3b06eee645618f04243666116c36 P a4336684d7414e61b17d557ddde64653 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } } peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } }
I20260807 07:02:25.096750 1366 leader_election.cc:288] T a03e3b06eee645618f04243666116c36 P a4336684d7414e61b17d557ddde64653 [CANDIDATE]: Term 1 election: Requested vote from peers d02e909ab20f4a12b000cb5ca64212e7 (127.21.83.3:35127), 32317a2a19304a51a248edde90974bbb (127.21.83.1:44117)
I20260807 07:02:25.097141 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03e3b06eee645618f04243666116c36" candidate_uuid: "a4336684d7414e61b17d557ddde64653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d02e909ab20f4a12b000cb5ca64212e7"
I20260807 07:02:25.097245 828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03e3b06eee645618f04243666116c36" candidate_uuid: "a4336684d7414e61b17d557ddde64653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32317a2a19304a51a248edde90974bbb"
I20260807 07:02:25.097319 1241 raft_consensus.cc:3037] T a03e3b06eee645618f04243666116c36 P d02e909ab20f4a12b000cb5ca64212e7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:02:25.097441 828 raft_consensus.cc:3037] T a03e3b06eee645618f04243666116c36 P 32317a2a19304a51a248edde90974bbb [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:02:25.099690 1241 raft_consensus.cc:2454] T a03e3b06eee645618f04243666116c36 P d02e909ab20f4a12b000cb5ca64212e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4336684d7414e61b17d557ddde64653 in term 1.
I20260807 07:02:25.099742 828 raft_consensus.cc:2454] T a03e3b06eee645618f04243666116c36 P 32317a2a19304a51a248edde90974bbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4336684d7414e61b17d557ddde64653 in term 1.
I20260807 07:02:25.100040 934 leader_election.cc:302] T a03e3b06eee645618f04243666116c36 P a4336684d7414e61b17d557ddde64653 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4336684d7414e61b17d557ddde64653, d02e909ab20f4a12b000cb5ca64212e7; no voters:
I20260807 07:02:25.100325 1366 raft_consensus.cc:2781] T a03e3b06eee645618f04243666116c36 P a4336684d7414e61b17d557ddde64653 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:02:25.100775 1366 raft_consensus.cc:686] T a03e3b06eee645618f04243666116c36 P a4336684d7414e61b17d557ddde64653 [term 1 LEADER]: Becoming Leader. State: Replica: a4336684d7414e61b17d557ddde64653, State: Running, Role: LEADER
I20260807 07:02:25.101245 1366 consensus_queue.cc:237] T a03e3b06eee645618f04243666116c36 P a4336684d7414e61b17d557ddde64653 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } } peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } }
I20260807 07:02:25.105883 604 catalog_manager.cc:5495] T a03e3b06eee645618f04243666116c36 P a4336684d7414e61b17d557ddde64653 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4336684d7414e61b17d557ddde64653 (127.21.83.2). New cstate: current_term: 1 leader_uuid: "a4336684d7414e61b17d557ddde64653" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } health_report { overall_health: UNKNOWN } } }
I20260807 07:02:25.169322 1348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:25.163661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41716 (local address 127.21.83.2:40429)
0807 07:02:25.163861 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:25.163868 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:25.164053 (+ 185us) server_negotiation.cc:407] Connection header received
0807 07:02:25.164287 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:25.164291 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:25.164374 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:25.164512 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:02:25.164884 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:25.165614 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:25.167105 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:25.167524 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:25.167651 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:25.168088 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:02:25.168659 (+ 571us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:02:25.168800 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:25.168938 (+ 138us) server_negotiation.cc:299] Negotiation successful
0807 07:02:25.169032 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 07:02:25.177735 1241 raft_consensus.cc:1261] T a03e3b06eee645618f04243666116c36 P d02e909ab20f4a12b000cb5ca64212e7 [term 1 FOLLOWER]: Refusing update from remote peer a4336684d7414e61b17d557ddde64653: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:02:25.177735 828 raft_consensus.cc:1261] T a03e3b06eee645618f04243666116c36 P 32317a2a19304a51a248edde90974bbb [term 1 FOLLOWER]: Refusing update from remote peer a4336684d7414e61b17d557ddde64653: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:02:25.178751 1366 consensus_queue.cc:1035] T a03e3b06eee645618f04243666116c36 P a4336684d7414e61b17d557ddde64653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:02:25.179065 1369 consensus_queue.cc:1035] T a03e3b06eee645618f04243666116c36 P a4336684d7414e61b17d557ddde64653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:02:25.189312 1373 mvcc.cc:205] Tried to move back new op lower bound from 7315808850637418496 to 7315808850338201600. Current Snapshot: MvccSnapshot[applied={T|T < 7315808850637418496}]
I20260807 07:02:25.190932 1376 mvcc.cc:205] Tried to move back new op lower bound from 7315808850637418496 to 7315808850338201600. Current Snapshot: MvccSnapshot[applied={T|T < 7315808850637418496}]
I20260807 07:02:25.205806 603 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:02:25.206049 603 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:02:25.209491 603 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0935abe2905a450e8cbcfa9aead19321: Sending DeleteTablet for 3 replicas of tablet a03e3b06eee645618f04243666116c36
I20260807 07:02:25.210537 1005 tablet_service.cc:1514] Processing DeleteTablet for tablet a03e3b06eee645618f04243666116c36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:02:25 UTC) from {username='oryx', principal='oryx/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:41708
I20260807 07:02:25.210676 1217 tablet_service.cc:1514] Processing DeleteTablet for tablet a03e3b06eee645618f04243666116c36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:02:25 UTC) from {username='oryx', principal='oryx/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:41542
I20260807 07:02:25.210999 799 tablet_service.cc:1514] Processing DeleteTablet for tablet a03e3b06eee645618f04243666116c36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:02:25 UTC) from {username='oryx', principal='oryx/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:58544
I20260807 07:02:25.211522 1389 tablet_replica.cc:330] T a03e3b06eee645618f04243666116c36 P a4336684d7414e61b17d557ddde64653: stopping tablet replica
I20260807 07:02:25.212023 1389 raft_consensus.cc:2229] T a03e3b06eee645618f04243666116c36 P a4336684d7414e61b17d557ddde64653 [term 1 LEADER]: Raft consensus shutting down.
I20260807 07:02:25.212507 603 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49768:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260807 07:02:25.213061 1389 raft_consensus.cc:2258] T a03e3b06eee645618f04243666116c36 P a4336684d7414e61b17d557ddde64653 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260807 07:02:25.213238 603 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 07:02:25.215670 1394 tablet_replica.cc:330] T a03e3b06eee645618f04243666116c36 P 32317a2a19304a51a248edde90974bbb: stopping tablet replica
I20260807 07:02:25.216184 1394 raft_consensus.cc:2229] T a03e3b06eee645618f04243666116c36 P 32317a2a19304a51a248edde90974bbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:02:25.216759 1394 raft_consensus.cc:2258] T a03e3b06eee645618f04243666116c36 P 32317a2a19304a51a248edde90974bbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:02:25.216802 1389 ts_tablet_manager.cc:1905] T a03e3b06eee645618f04243666116c36 P a4336684d7414e61b17d557ddde64653: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:02:25.217777 1392 tablet_replica.cc:330] T a03e3b06eee645618f04243666116c36 P d02e909ab20f4a12b000cb5ca64212e7: stopping tablet replica
I20260807 07:02:25.219566 1394 ts_tablet_manager.cc:1905] T a03e3b06eee645618f04243666116c36 P 32317a2a19304a51a248edde90974bbb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:02:25.220116 1392 raft_consensus.cc:2229] T a03e3b06eee645618f04243666116c36 P d02e909ab20f4a12b000cb5ca64212e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:02:25.220655 1392 raft_consensus.cc:2258] T a03e3b06eee645618f04243666116c36 P d02e909ab20f4a12b000cb5ca64212e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:02:25.224851 1217 tablet_service.cc:1467] Processing CreateTablet for tablet 3153cffbfeb74f29b9592722a81fe991 (DEFAULT_TABLE table=test-table [id=3d58f784cb3a4910a8545559dd01ee17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:02:25.224855 799 tablet_service.cc:1467] Processing CreateTablet for tablet 3153cffbfeb74f29b9592722a81fe991 (DEFAULT_TABLE table=test-table [id=3d58f784cb3a4910a8545559dd01ee17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:02:25.225360 799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3153cffbfeb74f29b9592722a81fe991. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:02:25.225353 1217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3153cffbfeb74f29b9592722a81fe991. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:02:25.227361 1392 ts_tablet_manager.cc:1905] T a03e3b06eee645618f04243666116c36 P d02e909ab20f4a12b000cb5ca64212e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:02:25.228968 1394 ts_tablet_manager.cc:1918] T a03e3b06eee645618f04243666116c36 P 32317a2a19304a51a248edde90974bbb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:02:25.229192 1394 log.cc:1196] T a03e3b06eee645618f04243666116c36 P 32317a2a19304a51a248edde90974bbb: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-0/wal/wals/a03e3b06eee645618f04243666116c36
I20260807 07:02:25.229271 1005 tablet_service.cc:1467] Processing CreateTablet for tablet 3153cffbfeb74f29b9592722a81fe991 (DEFAULT_TABLE table=test-table [id=3d58f784cb3a4910a8545559dd01ee17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:02:25.229905 1394 ts_tablet_manager.cc:1939] T a03e3b06eee645618f04243666116c36 P 32317a2a19304a51a248edde90974bbb: Deleting consensus metadata
I20260807 07:02:25.231917 1360 tablet_bootstrap.cc:492] T 3153cffbfeb74f29b9592722a81fe991 P d02e909ab20f4a12b000cb5ca64212e7: Bootstrap starting.
I20260807 07:02:25.231947 588 catalog_manager.cc:4841] TS 32317a2a19304a51a248edde90974bbb (127.21.83.1:44117): tablet a03e3b06eee645618f04243666116c36 (table test-table [id=4c3cf324684c4740807571abb332e107]) successfully deleted
I20260807 07:02:25.232477 1005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3153cffbfeb74f29b9592722a81fe991. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:02:25.233551 1360 tablet_bootstrap.cc:654] T 3153cffbfeb74f29b9592722a81fe991 P d02e909ab20f4a12b000cb5ca64212e7: Neither blocks nor log segments found. Creating new log.
I20260807 07:02:25.233933 1354 tablet_bootstrap.cc:492] T 3153cffbfeb74f29b9592722a81fe991 P 32317a2a19304a51a248edde90974bbb: Bootstrap starting.
I20260807 07:02:25.235562 1354 tablet_bootstrap.cc:654] T 3153cffbfeb74f29b9592722a81fe991 P 32317a2a19304a51a248edde90974bbb: Neither blocks nor log segments found. Creating new log.
I20260807 07:02:25.239418 1356 tablet_bootstrap.cc:492] T 3153cffbfeb74f29b9592722a81fe991 P a4336684d7414e61b17d557ddde64653: Bootstrap starting.
I20260807 07:02:25.241109 1356 tablet_bootstrap.cc:654] T 3153cffbfeb74f29b9592722a81fe991 P a4336684d7414e61b17d557ddde64653: Neither blocks nor log segments found. Creating new log.
I20260807 07:02:25.245414 1360 tablet_bootstrap.cc:492] T 3153cffbfeb74f29b9592722a81fe991 P d02e909ab20f4a12b000cb5ca64212e7: No bootstrap required, opened a new log
I20260807 07:02:25.245579 1360 ts_tablet_manager.cc:1397] T 3153cffbfeb74f29b9592722a81fe991 P d02e909ab20f4a12b000cb5ca64212e7: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.012s
I20260807 07:02:25.245842 1389 ts_tablet_manager.cc:1918] T a03e3b06eee645618f04243666116c36 P a4336684d7414e61b17d557ddde64653: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:02:25.246255 1389 log.cc:1196] T a03e3b06eee645618f04243666116c36 P a4336684d7414e61b17d557ddde64653: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-1/wal/wals/a03e3b06eee645618f04243666116c36
I20260807 07:02:25.246445 1360 raft_consensus.cc:348] T 3153cffbfeb74f29b9592722a81fe991 P d02e909ab20f4a12b000cb5ca64212e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } } peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } }
I20260807 07:02:25.246690 1360 raft_consensus.cc:374] T 3153cffbfeb74f29b9592722a81fe991 P d02e909ab20f4a12b000cb5ca64212e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:02:25.246804 1360 raft_consensus.cc:729] T 3153cffbfeb74f29b9592722a81fe991 P d02e909ab20f4a12b000cb5ca64212e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d02e909ab20f4a12b000cb5ca64212e7, State: Initialized, Role: FOLLOWER
I20260807 07:02:25.247143 1360 consensus_queue.cc:260] T 3153cffbfeb74f29b9592722a81fe991 P d02e909ab20f4a12b000cb5ca64212e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } } peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } }
I20260807 07:02:25.247723 1389 ts_tablet_manager.cc:1939] T a03e3b06eee645618f04243666116c36 P a4336684d7414e61b17d557ddde64653: Deleting consensus metadata
I20260807 07:02:25.249583 585 catalog_manager.cc:4841] TS a4336684d7414e61b17d557ddde64653 (127.21.83.2:40429): tablet a03e3b06eee645618f04243666116c36 (table test-table [id=4c3cf324684c4740807571abb332e107]) successfully deleted
I20260807 07:02:25.250110 1356 tablet_bootstrap.cc:492] T 3153cffbfeb74f29b9592722a81fe991 P a4336684d7414e61b17d557ddde64653: No bootstrap required, opened a new log
I20260807 07:02:25.250164 1360 ts_tablet_manager.cc:1428] T 3153cffbfeb74f29b9592722a81fe991 P d02e909ab20f4a12b000cb5ca64212e7: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260807 07:02:25.250273 1356 ts_tablet_manager.cc:1397] T 3153cffbfeb74f29b9592722a81fe991 P a4336684d7414e61b17d557ddde64653: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260807 07:02:25.251168 1356 raft_consensus.cc:348] T 3153cffbfeb74f29b9592722a81fe991 P a4336684d7414e61b17d557ddde64653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } } peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } }
I20260807 07:02:25.251399 1356 raft_consensus.cc:374] T 3153cffbfeb74f29b9592722a81fe991 P a4336684d7414e61b17d557ddde64653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:02:25.251516 1356 raft_consensus.cc:729] T 3153cffbfeb74f29b9592722a81fe991 P a4336684d7414e61b17d557ddde64653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4336684d7414e61b17d557ddde64653, State: Initialized, Role: FOLLOWER
I20260807 07:02:25.251824 1356 consensus_queue.cc:260] T 3153cffbfeb74f29b9592722a81fe991 P a4336684d7414e61b17d557ddde64653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } } peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } }
I20260807 07:02:25.252936 1356 ts_tablet_manager.cc:1428] T 3153cffbfeb74f29b9592722a81fe991 P a4336684d7414e61b17d557ddde64653: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260807 07:02:25.255137 1354 tablet_bootstrap.cc:492] T 3153cffbfeb74f29b9592722a81fe991 P 32317a2a19304a51a248edde90974bbb: No bootstrap required, opened a new log
I20260807 07:02:25.255321 1354 ts_tablet_manager.cc:1397] T 3153cffbfeb74f29b9592722a81fe991 P 32317a2a19304a51a248edde90974bbb: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.004s
I20260807 07:02:25.255831 1392 ts_tablet_manager.cc:1918] T a03e3b06eee645618f04243666116c36 P d02e909ab20f4a12b000cb5ca64212e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:02:25.256078 1392 log.cc:1196] T a03e3b06eee645618f04243666116c36 P d02e909ab20f4a12b000cb5ca64212e7: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-2/wal/wals/a03e3b06eee645618f04243666116c36
I20260807 07:02:25.256217 1354 raft_consensus.cc:348] T 3153cffbfeb74f29b9592722a81fe991 P 32317a2a19304a51a248edde90974bbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } } peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } }
I20260807 07:02:25.256481 1354 raft_consensus.cc:374] T 3153cffbfeb74f29b9592722a81fe991 P 32317a2a19304a51a248edde90974bbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:02:25.256600 1354 raft_consensus.cc:729] T 3153cffbfeb74f29b9592722a81fe991 P 32317a2a19304a51a248edde90974bbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32317a2a19304a51a248edde90974bbb, State: Initialized, Role: FOLLOWER
I20260807 07:02:25.256883 1392 ts_tablet_manager.cc:1939] T a03e3b06eee645618f04243666116c36 P d02e909ab20f4a12b000cb5ca64212e7: Deleting consensus metadata
I20260807 07:02:25.256879 1354 consensus_queue.cc:260] T 3153cffbfeb74f29b9592722a81fe991 P 32317a2a19304a51a248edde90974bbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } } peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } }
I20260807 07:02:25.257526 1354 ts_tablet_manager.cc:1428] T 3153cffbfeb74f29b9592722a81fe991 P 32317a2a19304a51a248edde90974bbb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260807 07:02:25.259073 585 catalog_manager.cc:4841] TS d02e909ab20f4a12b000cb5ca64212e7 (127.21.83.3:35127): tablet a03e3b06eee645618f04243666116c36 (table test-table [id=4c3cf324684c4740807571abb332e107]) successfully deleted
I20260807 07:02:25.510674 1366 raft_consensus.cc:482] T 3153cffbfeb74f29b9592722a81fe991 P a4336684d7414e61b17d557ddde64653 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:02:25.510910 1366 raft_consensus.cc:504] T 3153cffbfeb74f29b9592722a81fe991 P a4336684d7414e61b17d557ddde64653 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } } peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } }
I20260807 07:02:25.511564 1366 leader_election.cc:288] T 3153cffbfeb74f29b9592722a81fe991 P a4336684d7414e61b17d557ddde64653 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32317a2a19304a51a248edde90974bbb (127.21.83.1:44117), d02e909ab20f4a12b000cb5ca64212e7 (127.21.83.3:35127)
I20260807 07:02:25.512213 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3153cffbfeb74f29b9592722a81fe991" candidate_uuid: "a4336684d7414e61b17d557ddde64653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d02e909ab20f4a12b000cb5ca64212e7" is_pre_election: true
I20260807 07:02:25.512213 828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3153cffbfeb74f29b9592722a81fe991" candidate_uuid: "a4336684d7414e61b17d557ddde64653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32317a2a19304a51a248edde90974bbb" is_pre_election: true
I20260807 07:02:25.512485 828 raft_consensus.cc:2454] T 3153cffbfeb74f29b9592722a81fe991 P 32317a2a19304a51a248edde90974bbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4336684d7414e61b17d557ddde64653 in term 0.
I20260807 07:02:25.512485 1241 raft_consensus.cc:2454] T 3153cffbfeb74f29b9592722a81fe991 P d02e909ab20f4a12b000cb5ca64212e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4336684d7414e61b17d557ddde64653 in term 0.
I20260807 07:02:25.512977 934 leader_election.cc:302] T 3153cffbfeb74f29b9592722a81fe991 P a4336684d7414e61b17d557ddde64653 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4336684d7414e61b17d557ddde64653, d02e909ab20f4a12b000cb5ca64212e7; no voters:
I20260807 07:02:25.513273 1366 raft_consensus.cc:2781] T 3153cffbfeb74f29b9592722a81fe991 P a4336684d7414e61b17d557ddde64653 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 07:02:25.513427 1366 raft_consensus.cc:482] T 3153cffbfeb74f29b9592722a81fe991 P a4336684d7414e61b17d557ddde64653 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:02:25.513536 1366 raft_consensus.cc:3037] T 3153cffbfeb74f29b9592722a81fe991 P a4336684d7414e61b17d557ddde64653 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:02:25.516665 1366 raft_consensus.cc:504] T 3153cffbfeb74f29b9592722a81fe991 P a4336684d7414e61b17d557ddde64653 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } } peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } }
I20260807 07:02:25.517400 1366 leader_election.cc:288] T 3153cffbfeb74f29b9592722a81fe991 P a4336684d7414e61b17d557ddde64653 [CANDIDATE]: Term 1 election: Requested vote from peers 32317a2a19304a51a248edde90974bbb (127.21.83.1:44117), d02e909ab20f4a12b000cb5ca64212e7 (127.21.83.3:35127)
I20260807 07:02:25.517865 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3153cffbfeb74f29b9592722a81fe991" candidate_uuid: "a4336684d7414e61b17d557ddde64653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d02e909ab20f4a12b000cb5ca64212e7"
I20260807 07:02:25.517870 828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3153cffbfeb74f29b9592722a81fe991" candidate_uuid: "a4336684d7414e61b17d557ddde64653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32317a2a19304a51a248edde90974bbb"
I20260807 07:02:25.518069 1241 raft_consensus.cc:3037] T 3153cffbfeb74f29b9592722a81fe991 P d02e909ab20f4a12b000cb5ca64212e7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:02:25.518070 828 raft_consensus.cc:3037] T 3153cffbfeb74f29b9592722a81fe991 P 32317a2a19304a51a248edde90974bbb [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:02:25.520087 1241 raft_consensus.cc:2454] T 3153cffbfeb74f29b9592722a81fe991 P d02e909ab20f4a12b000cb5ca64212e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4336684d7414e61b17d557ddde64653 in term 1.
I20260807 07:02:25.520476 934 leader_election.cc:302] T 3153cffbfeb74f29b9592722a81fe991 P a4336684d7414e61b17d557ddde64653 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4336684d7414e61b17d557ddde64653, d02e909ab20f4a12b000cb5ca64212e7; no voters:
I20260807 07:02:25.520744 1366 raft_consensus.cc:2781] T 3153cffbfeb74f29b9592722a81fe991 P a4336684d7414e61b17d557ddde64653 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:02:25.520908 1366 raft_consensus.cc:686] T 3153cffbfeb74f29b9592722a81fe991 P a4336684d7414e61b17d557ddde64653 [term 1 LEADER]: Becoming Leader. State: Replica: a4336684d7414e61b17d557ddde64653, State: Running, Role: LEADER
I20260807 07:02:25.521008 828 raft_consensus.cc:2454] T 3153cffbfeb74f29b9592722a81fe991 P 32317a2a19304a51a248edde90974bbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4336684d7414e61b17d557ddde64653 in term 1.
I20260807 07:02:25.521201 1366 consensus_queue.cc:237] T 3153cffbfeb74f29b9592722a81fe991 P a4336684d7414e61b17d557ddde64653 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } } peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } }
I20260807 07:02:25.524147 603 catalog_manager.cc:5495] T 3153cffbfeb74f29b9592722a81fe991 P a4336684d7414e61b17d557ddde64653 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4336684d7414e61b17d557ddde64653 (127.21.83.2). New cstate: current_term: 1 leader_uuid: "a4336684d7414e61b17d557ddde64653" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } health_report { overall_health: UNKNOWN } } }
Aug 07 07:02:25 dist-test-slave-ncp9 krb5kdc[544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086142, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.83.62@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM
I20260807 07:02:25.645231 1404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:25.631657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:35983 (local address 127.0.0.1:49780)
0807 07:02:25.632418 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:25.632442 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:25.632573 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:25.632852 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:25.632855 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:25.633517 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:25.633741 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:25.633750 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:25.634795 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:25.634800 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:25.635757 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:25.635769 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:25.636443 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:25.636447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:25.636573 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:25.637319 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:25.637342 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:25.639611 (+ 2269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:25.642684 (+ 3073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:25.642694 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:25.642701 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:25.643120 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:25.643445 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:25.643448 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:25.643450 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:25.643608 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:25.644160 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:25.644170 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:25.644324 (+ 154us) client_negotiation.cc:769] Sending connection context
0807 07:02:25.644477 (+ 153us) client_negotiation.cc:240] Negotiation successful
0807 07:02:25.644749 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":396,"threads_started":1}
I20260807 07:02:25.645244 1406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:25.631691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49780 (local address 127.21.83.62:35983)
0807 07:02:25.632557 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:25.632561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:25.632585 (+ 24us) server_negotiation.cc:407] Connection header received
0807 07:02:25.632661 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:25.632665 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:25.632722 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:25.632821 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:25.633876 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:25.634668 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:25.635886 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:25.636280 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:25.636373 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:25.639827 (+ 3454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:25.639832 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:25.639835 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:25.639866 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:25.642553 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:25.643234 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:25.643239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:25.643241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:25.643364 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:25.643696 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:25.643699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:25.643701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:25.643957 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:25.644140 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:25.644585 (+ 445us) server_negotiation.cc:299] Negotiation successful
0807 07:02:25.644883 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":106,"server-negotiator.queue_time_us":741,"thread_start_us":572,"threads_started":1}
I20260807 07:02:25.650382 603 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:49780:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260807 07:02:25.651434 603 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 07:02:25.658357 799 tablet_service.cc:1467] Processing CreateTablet for tablet 549b9c8eade84a599fc4827cc4187e7e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d82ee4127cfd4f518c287f46b6b9d631]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 07:02:25.658439 1005 tablet_service.cc:1467] Processing CreateTablet for tablet 549b9c8eade84a599fc4827cc4187e7e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d82ee4127cfd4f518c287f46b6b9d631]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 07:02:25.658572 1217 tablet_service.cc:1467] Processing CreateTablet for tablet 549b9c8eade84a599fc4827cc4187e7e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d82ee4127cfd4f518c287f46b6b9d631]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 07:02:25.659003 1005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 549b9c8eade84a599fc4827cc4187e7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:02:25.659004 799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 549b9c8eade84a599fc4827cc4187e7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:02:25.659106 1217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 549b9c8eade84a599fc4827cc4187e7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:02:25.665257 1360 tablet_bootstrap.cc:492] T 549b9c8eade84a599fc4827cc4187e7e P d02e909ab20f4a12b000cb5ca64212e7: Bootstrap starting.
I20260807 07:02:25.666245 1356 tablet_bootstrap.cc:492] T 549b9c8eade84a599fc4827cc4187e7e P a4336684d7414e61b17d557ddde64653: Bootstrap starting.
I20260807 07:02:25.667081 1360 tablet_bootstrap.cc:654] T 549b9c8eade84a599fc4827cc4187e7e P d02e909ab20f4a12b000cb5ca64212e7: Neither blocks nor log segments found. Creating new log.
I20260807 07:02:25.667223 1354 tablet_bootstrap.cc:492] T 549b9c8eade84a599fc4827cc4187e7e P 32317a2a19304a51a248edde90974bbb: Bootstrap starting.
I20260807 07:02:25.667960 1356 tablet_bootstrap.cc:654] T 549b9c8eade84a599fc4827cc4187e7e P a4336684d7414e61b17d557ddde64653: Neither blocks nor log segments found. Creating new log.
I20260807 07:02:25.668874 1354 tablet_bootstrap.cc:654] T 549b9c8eade84a599fc4827cc4187e7e P 32317a2a19304a51a248edde90974bbb: Neither blocks nor log segments found. Creating new log.
I20260807 07:02:25.669533 1360 tablet_bootstrap.cc:492] T 549b9c8eade84a599fc4827cc4187e7e P d02e909ab20f4a12b000cb5ca64212e7: No bootstrap required, opened a new log
I20260807 07:02:25.669725 1360 ts_tablet_manager.cc:1397] T 549b9c8eade84a599fc4827cc4187e7e P d02e909ab20f4a12b000cb5ca64212e7: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260807 07:02:25.670205 1356 tablet_bootstrap.cc:492] T 549b9c8eade84a599fc4827cc4187e7e P a4336684d7414e61b17d557ddde64653: No bootstrap required, opened a new log
I20260807 07:02:25.670385 1356 ts_tablet_manager.cc:1397] T 549b9c8eade84a599fc4827cc4187e7e P a4336684d7414e61b17d557ddde64653: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260807 07:02:25.670578 1360 raft_consensus.cc:348] T 549b9c8eade84a599fc4827cc4187e7e P d02e909ab20f4a12b000cb5ca64212e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } } peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } }
I20260807 07:02:25.670867 1360 raft_consensus.cc:374] T 549b9c8eade84a599fc4827cc4187e7e P d02e909ab20f4a12b000cb5ca64212e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:02:25.670938 1360 raft_consensus.cc:729] T 549b9c8eade84a599fc4827cc4187e7e P d02e909ab20f4a12b000cb5ca64212e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d02e909ab20f4a12b000cb5ca64212e7, State: Initialized, Role: FOLLOWER
I20260807 07:02:25.671162 1356 raft_consensus.cc:348] T 549b9c8eade84a599fc4827cc4187e7e P a4336684d7414e61b17d557ddde64653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } } peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } }
I20260807 07:02:25.671442 1356 raft_consensus.cc:374] T 549b9c8eade84a599fc4827cc4187e7e P a4336684d7414e61b17d557ddde64653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:02:25.671536 1356 raft_consensus.cc:729] T 549b9c8eade84a599fc4827cc4187e7e P a4336684d7414e61b17d557ddde64653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4336684d7414e61b17d557ddde64653, State: Initialized, Role: FOLLOWER
I20260807 07:02:25.671263 1360 consensus_queue.cc:260] T 549b9c8eade84a599fc4827cc4187e7e P d02e909ab20f4a12b000cb5ca64212e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } } peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } }
I20260807 07:02:25.672297 1360 ts_tablet_manager.cc:1428] T 549b9c8eade84a599fc4827cc4187e7e P d02e909ab20f4a12b000cb5ca64212e7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260807 07:02:25.672207 1364 tablet_replica.cc:439] T 549b9c8eade84a599fc4827cc4187e7e P d02e909ab20f4a12b000cb5ca64212e7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } } peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } } }
I20260807 07:02:25.672446 1354 tablet_bootstrap.cc:492] T 549b9c8eade84a599fc4827cc4187e7e P 32317a2a19304a51a248edde90974bbb: No bootstrap required, opened a new log
I20260807 07:02:25.672564 1364 tablet_replica.cc:442] T 549b9c8eade84a599fc4827cc4187e7e P d02e909ab20f4a12b000cb5ca64212e7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 07:02:25.672616 1354 ts_tablet_manager.cc:1397] T 549b9c8eade84a599fc4827cc4187e7e P 32317a2a19304a51a248edde90974bbb: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260807 07:02:25.671799 1356 consensus_queue.cc:260] T 549b9c8eade84a599fc4827cc4187e7e P a4336684d7414e61b17d557ddde64653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } } peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } }
I20260807 07:02:25.673183 1356 ts_tablet_manager.cc:1428] T 549b9c8eade84a599fc4827cc4187e7e P a4336684d7414e61b17d557ddde64653: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260807 07:02:25.673044 1366 tablet_replica.cc:439] T 549b9c8eade84a599fc4827cc4187e7e P a4336684d7414e61b17d557ddde64653: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } } peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } } }
I20260807 07:02:25.673395 1366 tablet_replica.cc:442] T 549b9c8eade84a599fc4827cc4187e7e P a4336684d7414e61b17d557ddde64653: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 07:02:25.673389 1354 raft_consensus.cc:348] T 549b9c8eade84a599fc4827cc4187e7e P 32317a2a19304a51a248edde90974bbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } } peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } }
I20260807 07:02:25.673669 1354 raft_consensus.cc:374] T 549b9c8eade84a599fc4827cc4187e7e P 32317a2a19304a51a248edde90974bbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:02:25.673765 1354 raft_consensus.cc:729] T 549b9c8eade84a599fc4827cc4187e7e P 32317a2a19304a51a248edde90974bbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32317a2a19304a51a248edde90974bbb, State: Initialized, Role: FOLLOWER
I20260807 07:02:25.674049 1354 consensus_queue.cc:260] T 549b9c8eade84a599fc4827cc4187e7e P 32317a2a19304a51a248edde90974bbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } } peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } }
I20260807 07:02:25.674803 1354 ts_tablet_manager.cc:1428] T 549b9c8eade84a599fc4827cc4187e7e P 32317a2a19304a51a248edde90974bbb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260807 07:02:25.674551 1365 tablet_replica.cc:439] T 549b9c8eade84a599fc4827cc4187e7e P 32317a2a19304a51a248edde90974bbb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } } peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } } }
I20260807 07:02:25.674966 1365 tablet_replica.cc:442] T 549b9c8eade84a599fc4827cc4187e7e P 32317a2a19304a51a248edde90974bbb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 07:02:25.974229 1365 raft_consensus.cc:482] T 549b9c8eade84a599fc4827cc4187e7e P 32317a2a19304a51a248edde90974bbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:02:25.974494 1365 raft_consensus.cc:504] T 549b9c8eade84a599fc4827cc4187e7e P 32317a2a19304a51a248edde90974bbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } } peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } }
I20260807 07:02:25.976095 1365 leader_election.cc:288] T 549b9c8eade84a599fc4827cc4187e7e P 32317a2a19304a51a248edde90974bbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4336684d7414e61b17d557ddde64653 (127.21.83.2:40429), d02e909ab20f4a12b000cb5ca64212e7 (127.21.83.3:35127)
I20260807 07:02:25.976991 1366 raft_consensus.cc:482] T 549b9c8eade84a599fc4827cc4187e7e P a4336684d7414e61b17d557ddde64653 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:02:25.977208 1366 raft_consensus.cc:504] T 549b9c8eade84a599fc4827cc4187e7e P a4336684d7414e61b17d557ddde64653 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } } peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } }
I20260807 07:02:25.977914 1366 leader_election.cc:288] T 549b9c8eade84a599fc4827cc4187e7e P a4336684d7414e61b17d557ddde64653 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32317a2a19304a51a248edde90974bbb (127.21.83.1:44117), d02e909ab20f4a12b000cb5ca64212e7 (127.21.83.3:35127)
I20260807 07:02:25.978803 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "549b9c8eade84a599fc4827cc4187e7e" candidate_uuid: "a4336684d7414e61b17d557ddde64653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d02e909ab20f4a12b000cb5ca64212e7" is_pre_election: true
I20260807 07:02:25.979135 1241 raft_consensus.cc:2454] T 549b9c8eade84a599fc4827cc4187e7e P d02e909ab20f4a12b000cb5ca64212e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4336684d7414e61b17d557ddde64653 in term 0.
I20260807 07:02:25.979741 934 leader_election.cc:302] T 549b9c8eade84a599fc4827cc4187e7e P a4336684d7414e61b17d557ddde64653 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4336684d7414e61b17d557ddde64653, d02e909ab20f4a12b000cb5ca64212e7; no voters:
I20260807 07:02:25.980633 828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "549b9c8eade84a599fc4827cc4187e7e" candidate_uuid: "a4336684d7414e61b17d557ddde64653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32317a2a19304a51a248edde90974bbb" is_pre_election: true
I20260807 07:02:25.980970 828 raft_consensus.cc:2454] T 549b9c8eade84a599fc4827cc4187e7e P 32317a2a19304a51a248edde90974bbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4336684d7414e61b17d557ddde64653 in term 0.
I20260807 07:02:25.981681 1366 raft_consensus.cc:2781] T 549b9c8eade84a599fc4827cc4187e7e P a4336684d7414e61b17d557ddde64653 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 07:02:25.981797 1366 raft_consensus.cc:482] T 549b9c8eade84a599fc4827cc4187e7e P a4336684d7414e61b17d557ddde64653 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:02:25.981909 1366 raft_consensus.cc:3037] T 549b9c8eade84a599fc4827cc4187e7e P a4336684d7414e61b17d557ddde64653 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:02:25.984537 1411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:25.976347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.2:40429 (local address 127.21.83.1:58131)
0807 07:02:25.976975 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:25.976999 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:25.977187 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:25.977801 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:25.977807 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:25.977836 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:02:25.978175 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:25.978190 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:25.979699 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:25.979705 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:25.981826 (+ 2121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:25.981844 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:25.983640 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:25.983647 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:25.983848 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:25.983888 (+ 40us) client_negotiation.cc:769] Sending connection context
0807 07:02:25.983989 (+ 101us) client_negotiation.cc:240] Negotiation successful
0807 07:02:25.984145 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":314,"threads_started":1}
I20260807 07:02:25.985301 1366 raft_consensus.cc:504] T 549b9c8eade84a599fc4827cc4187e7e P a4336684d7414e61b17d557ddde64653 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } } peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } }
I20260807 07:02:25.985916 1412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:25.976459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:58131 (local address 127.21.83.2:40429)
0807 07:02:25.976999 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:25.977005 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:25.977313 (+ 308us) server_negotiation.cc:407] Connection header received
0807 07:02:25.977393 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:25.977398 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:25.977474 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:25.977585 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:02:25.978368 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:25.979500 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:25.982034 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:25.983249 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:25.983397 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:25.983525 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:25.985394 (+ 1869us) server_negotiation.cc:299] Negotiation successful
0807 07:02:25.985566 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":266,"threads_started":1}
I20260807 07:02:25.986539 828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "549b9c8eade84a599fc4827cc4187e7e" candidate_uuid: "a4336684d7414e61b17d557ddde64653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32317a2a19304a51a248edde90974bbb"
I20260807 07:02:25.986830 828 raft_consensus.cc:3037] T 549b9c8eade84a599fc4827cc4187e7e P 32317a2a19304a51a248edde90974bbb [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:02:25.986831 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "549b9c8eade84a599fc4827cc4187e7e" candidate_uuid: "a4336684d7414e61b17d557ddde64653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d02e909ab20f4a12b000cb5ca64212e7"
I20260807 07:02:25.987076 1241 raft_consensus.cc:3037] T 549b9c8eade84a599fc4827cc4187e7e P d02e909ab20f4a12b000cb5ca64212e7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:02:25.990089 828 raft_consensus.cc:2454] T 549b9c8eade84a599fc4827cc4187e7e P 32317a2a19304a51a248edde90974bbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4336684d7414e61b17d557ddde64653 in term 1.
I20260807 07:02:25.990089 1241 raft_consensus.cc:2454] T 549b9c8eade84a599fc4827cc4187e7e P d02e909ab20f4a12b000cb5ca64212e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4336684d7414e61b17d557ddde64653 in term 1.
I20260807 07:02:25.990662 934 leader_election.cc:302] T 549b9c8eade84a599fc4827cc4187e7e P a4336684d7414e61b17d557ddde64653 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4336684d7414e61b17d557ddde64653, d02e909ab20f4a12b000cb5ca64212e7; no voters:
I20260807 07:02:25.990765 1366 leader_election.cc:288] T 549b9c8eade84a599fc4827cc4187e7e P a4336684d7414e61b17d557ddde64653 [CANDIDATE]: Term 1 election: Requested vote from peers 32317a2a19304a51a248edde90974bbb (127.21.83.1:44117), d02e909ab20f4a12b000cb5ca64212e7 (127.21.83.3:35127)
I20260807 07:02:25.991066 1366 raft_consensus.cc:2781] T 549b9c8eade84a599fc4827cc4187e7e P a4336684d7414e61b17d557ddde64653 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:02:25.991250 1366 raft_consensus.cc:686] T 549b9c8eade84a599fc4827cc4187e7e P a4336684d7414e61b17d557ddde64653 [term 1 LEADER]: Becoming Leader. State: Replica: a4336684d7414e61b17d557ddde64653, State: Running, Role: LEADER
I20260807 07:02:25.991521 1366 consensus_queue.cc:237] T 549b9c8eade84a599fc4827cc4187e7e P a4336684d7414e61b17d557ddde64653 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } } peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } }
I20260807 07:02:25.992290 1039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "549b9c8eade84a599fc4827cc4187e7e" candidate_uuid: "32317a2a19304a51a248edde90974bbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4336684d7414e61b17d557ddde64653" is_pre_election: true
I20260807 07:02:25.993273 1369 tablet_replica.cc:439] T 549b9c8eade84a599fc4827cc4187e7e P a4336684d7414e61b17d557ddde64653: TxnStatusTablet state changed. Reason: New leader a4336684d7414e61b17d557ddde64653. Latest consensus state: current_term: 1 leader_uuid: "a4336684d7414e61b17d557ddde64653" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } } peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } } }
I20260807 07:02:25.993587 1369 tablet_replica.cc:442] T 549b9c8eade84a599fc4827cc4187e7e P a4336684d7414e61b17d557ddde64653: This TxnStatusTablet replica's current role is: LEADER
I20260807 07:02:25.994997 604 catalog_manager.cc:5495] T 549b9c8eade84a599fc4827cc4187e7e P a4336684d7414e61b17d557ddde64653 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4336684d7414e61b17d557ddde64653 (127.21.83.2). New cstate: current_term: 1 leader_uuid: "a4336684d7414e61b17d557ddde64653" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } health_report { overall_health: UNKNOWN } } }
I20260807 07:02:25.995326 1418 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 07:02:25.995561 1418 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260807 07:02:25.998056 1411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:25.977206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.3:35127 (local address 127.21.83.1:60915)
0807 07:02:25.985080 (+ 7874us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:25.985106 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:25.985239 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:25.991714 (+ 6475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:25.991719 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:25.991840 (+ 121us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:02:25.992053 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:25.992068 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:25.993847 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:25.993853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:25.995686 (+ 1833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:25.995706 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:25.997200 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:25.997207 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:25.997415 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:25.997454 (+ 39us) client_negotiation.cc:769] Sending connection context
0807 07:02:25.997572 (+ 118us) client_negotiation.cc:240] Negotiation successful
0807 07:02:25.997741 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7487,"thread_start_us":13909,"threads_started":1}
I20260807 07:02:25.999038 1415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:25.977403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:60915 (local address 127.21.83.3:35127)
0807 07:02:25.991221 (+ 13818us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:25.991228 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:25.991253 (+ 25us) server_negotiation.cc:407] Connection header received
0807 07:02:25.991331 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:25.991337 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:25.991422 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:25.991543 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:02:25.992242 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:25.993624 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:25.995874 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:25.996762 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:25.996925 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:25.997060 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:25.998525 (+ 1465us) server_negotiation.cc:299] Negotiation successful
0807 07:02:25.998693 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13651,"thread_start_us":13453,"threads_started":1}
I20260807 07:02:26.000036 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "549b9c8eade84a599fc4827cc4187e7e" candidate_uuid: "32317a2a19304a51a248edde90974bbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d02e909ab20f4a12b000cb5ca64212e7" is_pre_election: true
I20260807 07:02:26.000349 1241 raft_consensus.cc:2379] T 549b9c8eade84a599fc4827cc4187e7e P d02e909ab20f4a12b000cb5ca64212e7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32317a2a19304a51a248edde90974bbb in current term 1: Already voted for candidate a4336684d7414e61b17d557ddde64653 in this term.
I20260807 07:02:26.001046 727 leader_election.cc:302] T 549b9c8eade84a599fc4827cc4187e7e P 32317a2a19304a51a248edde90974bbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32317a2a19304a51a248edde90974bbb; no voters: a4336684d7414e61b17d557ddde64653, d02e909ab20f4a12b000cb5ca64212e7
I20260807 07:02:26.001485 1365 raft_consensus.cc:2726] T 549b9c8eade84a599fc4827cc4187e7e P 32317a2a19304a51a248edde90974bbb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 07:02:26.054201 1369 consensus_queue.cc:1035] T 3153cffbfeb74f29b9592722a81fe991 P a4336684d7414e61b17d557ddde64653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:02:26.054557 1404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:26.049087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.2:40429 (local address 127.0.0.1:41718)
0807 07:02:26.049328 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:26.049345 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:26.049504 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:26.050021 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:26.050025 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:26.050039 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 07:02:26.050227 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:26.050237 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:26.051241 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:26.051247 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:26.052272 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:26.052284 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:26.053149 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:26.053156 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:26.053370 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:26.053455 (+ 85us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 07:02:26.053956 (+ 501us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 07:02:26.053981 (+ 25us) client_negotiation.cc:769] Sending connection context
0807 07:02:26.054039 (+ 58us) client_negotiation.cc:240] Negotiation successful
0807 07:02:26.054215 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260807 07:02:26.055365 1412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:26.049028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41718 (local address 127.21.83.2:40429)
0807 07:02:26.049227 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:26.049233 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:26.049419 (+ 186us) server_negotiation.cc:407] Connection header received
0807 07:02:26.049813 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:26.049817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:26.049891 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:26.049975 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:02:26.050365 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:26.051097 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:26.052415 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:26.052985 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:26.053106 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:26.053660 (+ 554us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:02:26.053811 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:02:26.054956 (+ 1145us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:26.055016 (+ 60us) server_negotiation.cc:299] Negotiation successful
0807 07:02:26.055103 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 07:02:26.060746 1241 raft_consensus.cc:1261] T 549b9c8eade84a599fc4827cc4187e7e P d02e909ab20f4a12b000cb5ca64212e7 [term 1 FOLLOWER]: Refusing update from remote peer a4336684d7414e61b17d557ddde64653: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:02:26.060765 828 raft_consensus.cc:1261] T 549b9c8eade84a599fc4827cc4187e7e P 32317a2a19304a51a248edde90974bbb [term 1 FOLLOWER]: Refusing update from remote peer a4336684d7414e61b17d557ddde64653: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:02:26.064844 1430 consensus_queue.cc:1035] T 549b9c8eade84a599fc4827cc4187e7e P a4336684d7414e61b17d557ddde64653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260807 07:02:26.064882 1427 consensus_queue.cc:1035] T 3153cffbfeb74f29b9592722a81fe991 P a4336684d7414e61b17d557ddde64653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260807 07:02:26.066141 1366 consensus_queue.cc:1035] T 549b9c8eade84a599fc4827cc4187e7e P a4336684d7414e61b17d557ddde64653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:02:26.079257 1364 tablet_replica.cc:439] T 549b9c8eade84a599fc4827cc4187e7e P d02e909ab20f4a12b000cb5ca64212e7: TxnStatusTablet state changed. Reason: New leader a4336684d7414e61b17d557ddde64653. Latest consensus state: current_term: 1 leader_uuid: "a4336684d7414e61b17d557ddde64653" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } } peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } } }
I20260807 07:02:26.079625 1364 tablet_replica.cc:442] T 549b9c8eade84a599fc4827cc4187e7e P d02e909ab20f4a12b000cb5ca64212e7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 07:02:26.088433 1366 tablet_replica.cc:439] T 549b9c8eade84a599fc4827cc4187e7e P a4336684d7414e61b17d557ddde64653: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a4336684d7414e61b17d557ddde64653" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } } peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } } }
I20260807 07:02:26.089115 1366 tablet_replica.cc:442] T 549b9c8eade84a599fc4827cc4187e7e P a4336684d7414e61b17d557ddde64653: This TxnStatusTablet replica's current role is: LEADER
I20260807 07:02:26.094051 1364 tablet_replica.cc:439] T 549b9c8eade84a599fc4827cc4187e7e P d02e909ab20f4a12b000cb5ca64212e7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a4336684d7414e61b17d557ddde64653" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } } peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } } }
I20260807 07:02:26.094404 1364 tablet_replica.cc:442] T 549b9c8eade84a599fc4827cc4187e7e P d02e909ab20f4a12b000cb5ca64212e7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 07:02:26.095315 1365 tablet_replica.cc:439] T 549b9c8eade84a599fc4827cc4187e7e P 32317a2a19304a51a248edde90974bbb: TxnStatusTablet state changed. Reason: New leader a4336684d7414e61b17d557ddde64653. Latest consensus state: current_term: 1 leader_uuid: "a4336684d7414e61b17d557ddde64653" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } } peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } } }
I20260807 07:02:26.095743 1365 tablet_replica.cc:442] T 549b9c8eade84a599fc4827cc4187e7e P 32317a2a19304a51a248edde90974bbb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 07:02:26.097316 1427 tablet_replica.cc:439] T 549b9c8eade84a599fc4827cc4187e7e P a4336684d7414e61b17d557ddde64653: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a4336684d7414e61b17d557ddde64653" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } } peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } } }
I20260807 07:02:26.097747 1427 tablet_replica.cc:442] T 549b9c8eade84a599fc4827cc4187e7e P a4336684d7414e61b17d557ddde64653: This TxnStatusTablet replica's current role is: LEADER
I20260807 07:02:26.108014 1365 tablet_replica.cc:439] T 549b9c8eade84a599fc4827cc4187e7e P 32317a2a19304a51a248edde90974bbb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a4336684d7414e61b17d557ddde64653" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } } peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } } }
I20260807 07:02:26.108397 1365 tablet_replica.cc:442] T 549b9c8eade84a599fc4827cc4187e7e P 32317a2a19304a51a248edde90974bbb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 07:02:26.129381 1412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:26.121296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:36953 (local address 127.21.83.2:40429)
0807 07:02:26.121492 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:26.121497 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:26.122271 (+ 774us) server_negotiation.cc:407] Connection header received
0807 07:02:26.122515 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:26.122521 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:26.122603 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:26.122710 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:02:26.123473 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:26.124764 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:26.127085 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:26.127812 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:26.127950 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:26.128496 (+ 546us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:02:26.128648 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:02:26.128729 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:26.128874 (+ 145us) server_negotiation.cc:299] Negotiation successful
0807 07:02:26.129004 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 07:02:26.129524 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:26.121322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.2:40429 (local address 127.21.83.2:36953)
0807 07:02:26.122127 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:26.122148 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:26.122346 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:26.122862 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:26.122866 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:26.122878 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 07:02:26.123278 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:26.123290 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:26.124943 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:26.124949 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:26.126920 (+ 1971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:26.126937 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:26.128008 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:26.128014 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:26.128207 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:26.128290 (+ 83us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 07:02:26.128750 (+ 460us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 07:02:26.128762 (+ 12us) client_negotiation.cc:769] Sending connection context
0807 07:02:26.128832 (+ 70us) client_negotiation.cc:240] Negotiation successful
0807 07:02:26.129050 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":422,"threads_started":1}
I20260807 07:02:26.205578 604 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:02:26.205895 604 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:02:26.209091 604 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0935abe2905a450e8cbcfa9aead19321: Sending DeleteTablet for 3 replicas of tablet 3153cffbfeb74f29b9592722a81fe991
I20260807 07:02:26.209952 799 tablet_service.cc:1514] Processing DeleteTablet for tablet 3153cffbfeb74f29b9592722a81fe991 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:02:26 UTC) from {username='oryx', principal='oryx/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:58544
I20260807 07:02:26.210340 1217 tablet_service.cc:1514] Processing DeleteTablet for tablet 3153cffbfeb74f29b9592722a81fe991 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:02:26 UTC) from {username='oryx', principal='oryx/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:41542
I20260807 07:02:26.211129 1001 tablet_service.cc:1514] Processing DeleteTablet for tablet 3153cffbfeb74f29b9592722a81fe991 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:02:26 UTC) from {username='oryx', principal='oryx/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:41708
I20260807 07:02:26.219951 1451 tablet_replica.cc:330] T 3153cffbfeb74f29b9592722a81fe991 P a4336684d7414e61b17d557ddde64653: stopping tablet replica
I20260807 07:02:26.220247 1451 raft_consensus.cc:2229] T 3153cffbfeb74f29b9592722a81fe991 P a4336684d7414e61b17d557ddde64653 [term 1 LEADER]: Raft consensus shutting down.
I20260807 07:02:26.220745 1451 raft_consensus.cc:2258] T 3153cffbfeb74f29b9592722a81fe991 P a4336684d7414e61b17d557ddde64653 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:02:26.223565 1451 ts_tablet_manager.cc:1905] T 3153cffbfeb74f29b9592722a81fe991 P a4336684d7414e61b17d557ddde64653: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:02:26.226742 1450 tablet_replica.cc:330] T 3153cffbfeb74f29b9592722a81fe991 P d02e909ab20f4a12b000cb5ca64212e7: stopping tablet replica
I20260807 07:02:26.227144 1450 raft_consensus.cc:2229] T 3153cffbfeb74f29b9592722a81fe991 P d02e909ab20f4a12b000cb5ca64212e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:02:26.227537 1450 raft_consensus.cc:2258] T 3153cffbfeb74f29b9592722a81fe991 P d02e909ab20f4a12b000cb5ca64212e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:02:26.228551 1450 ts_tablet_manager.cc:1905] T 3153cffbfeb74f29b9592722a81fe991 P d02e909ab20f4a12b000cb5ca64212e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:02:26.233915 1450 ts_tablet_manager.cc:1918] T 3153cffbfeb74f29b9592722a81fe991 P d02e909ab20f4a12b000cb5ca64212e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 07:02:26.234057 1450 log.cc:1196] T 3153cffbfeb74f29b9592722a81fe991 P d02e909ab20f4a12b000cb5ca64212e7: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-2/wal/wals/3153cffbfeb74f29b9592722a81fe991
I20260807 07:02:26.234526 1450 ts_tablet_manager.cc:1939] T 3153cffbfeb74f29b9592722a81fe991 P d02e909ab20f4a12b000cb5ca64212e7: Deleting consensus metadata
I20260807 07:02:26.235129 1451 ts_tablet_manager.cc:1918] T 3153cffbfeb74f29b9592722a81fe991 P a4336684d7414e61b17d557ddde64653: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 07:02:26.235266 1451 log.cc:1196] T 3153cffbfeb74f29b9592722a81fe991 P a4336684d7414e61b17d557ddde64653: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-1/wal/wals/3153cffbfeb74f29b9592722a81fe991
I20260807 07:02:26.235695 1451 ts_tablet_manager.cc:1939] T 3153cffbfeb74f29b9592722a81fe991 P a4336684d7414e61b17d557ddde64653: Deleting consensus metadata
I20260807 07:02:26.235756 585 catalog_manager.cc:4841] TS d02e909ab20f4a12b000cb5ca64212e7 (127.21.83.3:35127): tablet 3153cffbfeb74f29b9592722a81fe991 (table test-table [id=3d58f784cb3a4910a8545559dd01ee17]) successfully deleted
I20260807 07:02:26.236832 585 catalog_manager.cc:4841] TS a4336684d7414e61b17d557ddde64653 (127.21.83.2:40429): tablet 3153cffbfeb74f29b9592722a81fe991 (table test-table [id=3d58f784cb3a4910a8545559dd01ee17]) successfully deleted
I20260807 07:02:26.238127 1449 tablet_replica.cc:330] T 3153cffbfeb74f29b9592722a81fe991 P 32317a2a19304a51a248edde90974bbb: stopping tablet replica
I20260807 07:02:26.238328 1449 raft_consensus.cc:2229] T 3153cffbfeb74f29b9592722a81fe991 P 32317a2a19304a51a248edde90974bbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:02:26.238560 1449 raft_consensus.cc:2258] T 3153cffbfeb74f29b9592722a81fe991 P 32317a2a19304a51a248edde90974bbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:02:26.239640 1449 ts_tablet_manager.cc:1905] T 3153cffbfeb74f29b9592722a81fe991 P 32317a2a19304a51a248edde90974bbb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:02:26.247107 1449 ts_tablet_manager.cc:1918] T 3153cffbfeb74f29b9592722a81fe991 P 32317a2a19304a51a248edde90974bbb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 07:02:26.247287 1449 log.cc:1196] T 3153cffbfeb74f29b9592722a81fe991 P 32317a2a19304a51a248edde90974bbb: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-0/wal/wals/3153cffbfeb74f29b9592722a81fe991
I20260807 07:02:26.247749 1449 ts_tablet_manager.cc:1939] T 3153cffbfeb74f29b9592722a81fe991 P 32317a2a19304a51a248edde90974bbb: Deleting consensus metadata
I20260807 07:02:26.248991 588 catalog_manager.cc:4841] TS 32317a2a19304a51a248edde90974bbb (127.21.83.1:44117): tablet 3153cffbfeb74f29b9592722a81fe991 (table test-table [id=3d58f784cb3a4910a8545559dd01ee17]) successfully deleted
Aug 07 07:02:26 dist-test-slave-ncp9 krb5kdc[544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 07:02:26 dist-test-slave-ncp9 krb5kdc[544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:26 dist-test-slave-ncp9 krb5kdc[544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
W20260807 07:02:26.299201 1461 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:26.282857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49782 (local address 127.21.83.62:35983)
0807 07:02:26.283703 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:26.283708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:26.283728 (+ 20us) server_negotiation.cc:407] Connection header received
0807 07:02:26.283787 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:26.283791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:26.283851 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:26.283945 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:26.286812 (+ 2867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:26.287996 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:26.290553 (+ 2557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:26.290995 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:26.291119 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:26.298779 (+ 7660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49782: BlockingRecv error: recv got EOF from 127.0.0.1:49782 (error 108)
Metrics: {"server-negotiator.queue_time_us":721,"thread_start_us":592,"threads_started":1}
Aug 07 07:02:26 dist-test-slave-ncp9 krb5kdc[544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM
I20260807 07:02:26.325253 1461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:26.306259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49796 (local address 127.21.83.62:35983)
0807 07:02:26.306479 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:26.306486 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:26.306829 (+ 343us) server_negotiation.cc:407] Connection header received
0807 07:02:26.306972 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:26.306981 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:26.307074 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:26.307205 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:26.308477 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:26.309680 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:26.311299 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:26.312416 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:26.312597 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:26.317730 (+ 5133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:26.317735 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:26.317738 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:26.317775 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:26.321520 (+ 3745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:26.322409 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:26.322413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:26.322416 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:26.322598 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:26.323089 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:26.323094 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:26.323097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:26.323483 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:26.323768 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:26.324167 (+ 399us) server_negotiation.cc:299] Negotiation successful
0807 07:02:26.324387 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 07:02:26.330685 604 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49796:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 07:02:26.331432 604 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 07:02:26.339982 799 tablet_service.cc:1467] Processing CreateTablet for tablet d55cc40d376e47b2ab531e102b8d28c0 (DEFAULT_TABLE table=test-table [id=f2e26e0c98bf4d7f891a5bdd1b86b3fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:02:26.340240 1217 tablet_service.cc:1467] Processing CreateTablet for tablet d55cc40d376e47b2ab531e102b8d28c0 (DEFAULT_TABLE table=test-table [id=f2e26e0c98bf4d7f891a5bdd1b86b3fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:02:26.340588 799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d55cc40d376e47b2ab531e102b8d28c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:02:26.340756 1217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d55cc40d376e47b2ab531e102b8d28c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:02:26.344574 1001 tablet_service.cc:1467] Processing CreateTablet for tablet d55cc40d376e47b2ab531e102b8d28c0 (DEFAULT_TABLE table=test-table [id=f2e26e0c98bf4d7f891a5bdd1b86b3fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:02:26.345780 1001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d55cc40d376e47b2ab531e102b8d28c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:02:26.355997 1470 tablet_bootstrap.cc:492] T d55cc40d376e47b2ab531e102b8d28c0 P d02e909ab20f4a12b000cb5ca64212e7: Bootstrap starting.
I20260807 07:02:26.357014 1471 tablet_bootstrap.cc:492] T d55cc40d376e47b2ab531e102b8d28c0 P 32317a2a19304a51a248edde90974bbb: Bootstrap starting.
I20260807 07:02:26.359440 1470 tablet_bootstrap.cc:654] T d55cc40d376e47b2ab531e102b8d28c0 P d02e909ab20f4a12b000cb5ca64212e7: Neither blocks nor log segments found. Creating new log.
I20260807 07:02:26.360496 1471 tablet_bootstrap.cc:654] T d55cc40d376e47b2ab531e102b8d28c0 P 32317a2a19304a51a248edde90974bbb: Neither blocks nor log segments found. Creating new log.
I20260807 07:02:26.368054 1471 tablet_bootstrap.cc:492] T d55cc40d376e47b2ab531e102b8d28c0 P 32317a2a19304a51a248edde90974bbb: No bootstrap required, opened a new log
I20260807 07:02:26.368306 1471 ts_tablet_manager.cc:1397] T d55cc40d376e47b2ab531e102b8d28c0 P 32317a2a19304a51a248edde90974bbb: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20260807 07:02:26.369650 1471 raft_consensus.cc:348] T d55cc40d376e47b2ab531e102b8d28c0 P 32317a2a19304a51a248edde90974bbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } } peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } }
I20260807 07:02:26.370020 1471 raft_consensus.cc:374] T d55cc40d376e47b2ab531e102b8d28c0 P 32317a2a19304a51a248edde90974bbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:02:26.370179 1471 raft_consensus.cc:729] T d55cc40d376e47b2ab531e102b8d28c0 P 32317a2a19304a51a248edde90974bbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32317a2a19304a51a248edde90974bbb, State: Initialized, Role: FOLLOWER
I20260807 07:02:26.370541 1471 consensus_queue.cc:260] T d55cc40d376e47b2ab531e102b8d28c0 P 32317a2a19304a51a248edde90974bbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } } peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } }
I20260807 07:02:26.374428 1471 ts_tablet_manager.cc:1428] T d55cc40d376e47b2ab531e102b8d28c0 P 32317a2a19304a51a248edde90974bbb: Time spent starting tablet: real 0.006s user 0.003s sys 0.001s
I20260807 07:02:26.377506 1472 tablet_bootstrap.cc:492] T d55cc40d376e47b2ab531e102b8d28c0 P a4336684d7414e61b17d557ddde64653: Bootstrap starting.
I20260807 07:02:26.380122 1470 tablet_bootstrap.cc:492] T d55cc40d376e47b2ab531e102b8d28c0 P d02e909ab20f4a12b000cb5ca64212e7: No bootstrap required, opened a new log
I20260807 07:02:26.380409 1470 ts_tablet_manager.cc:1397] T d55cc40d376e47b2ab531e102b8d28c0 P d02e909ab20f4a12b000cb5ca64212e7: Time spent bootstrapping tablet: real 0.024s user 0.014s sys 0.008s
I20260807 07:02:26.380685 1472 tablet_bootstrap.cc:654] T d55cc40d376e47b2ab531e102b8d28c0 P a4336684d7414e61b17d557ddde64653: Neither blocks nor log segments found. Creating new log.
I20260807 07:02:26.381755 1470 raft_consensus.cc:348] T d55cc40d376e47b2ab531e102b8d28c0 P d02e909ab20f4a12b000cb5ca64212e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } } peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } }
I20260807 07:02:26.382082 1470 raft_consensus.cc:374] T d55cc40d376e47b2ab531e102b8d28c0 P d02e909ab20f4a12b000cb5ca64212e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:02:26.382339 1470 raft_consensus.cc:729] T d55cc40d376e47b2ab531e102b8d28c0 P d02e909ab20f4a12b000cb5ca64212e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d02e909ab20f4a12b000cb5ca64212e7, State: Initialized, Role: FOLLOWER
I20260807 07:02:26.382757 1470 consensus_queue.cc:260] T d55cc40d376e47b2ab531e102b8d28c0 P d02e909ab20f4a12b000cb5ca64212e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } } peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } }
I20260807 07:02:26.384231 1470 ts_tablet_manager.cc:1428] T d55cc40d376e47b2ab531e102b8d28c0 P d02e909ab20f4a12b000cb5ca64212e7: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260807 07:02:26.391880 1472 tablet_bootstrap.cc:492] T d55cc40d376e47b2ab531e102b8d28c0 P a4336684d7414e61b17d557ddde64653: No bootstrap required, opened a new log
I20260807 07:02:26.392179 1472 ts_tablet_manager.cc:1397] T d55cc40d376e47b2ab531e102b8d28c0 P a4336684d7414e61b17d557ddde64653: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20260807 07:02:26.393667 1472 raft_consensus.cc:348] T d55cc40d376e47b2ab531e102b8d28c0 P a4336684d7414e61b17d557ddde64653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } } peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } }
I20260807 07:02:26.394025 1472 raft_consensus.cc:374] T d55cc40d376e47b2ab531e102b8d28c0 P a4336684d7414e61b17d557ddde64653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:02:26.394200 1472 raft_consensus.cc:729] T d55cc40d376e47b2ab531e102b8d28c0 P a4336684d7414e61b17d557ddde64653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4336684d7414e61b17d557ddde64653, State: Initialized, Role: FOLLOWER
I20260807 07:02:26.394526 1472 consensus_queue.cc:260] T d55cc40d376e47b2ab531e102b8d28c0 P a4336684d7414e61b17d557ddde64653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } } peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } }
I20260807 07:02:26.395542 1472 ts_tablet_manager.cc:1428] T d55cc40d376e47b2ab531e102b8d28c0 P a4336684d7414e61b17d557ddde64653: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260807 07:02:26.547438 1364 raft_consensus.cc:482] T d55cc40d376e47b2ab531e102b8d28c0 P d02e909ab20f4a12b000cb5ca64212e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:02:26.547945 1364 raft_consensus.cc:504] T d55cc40d376e47b2ab531e102b8d28c0 P d02e909ab20f4a12b000cb5ca64212e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } } peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } }
I20260807 07:02:26.549564 1364 leader_election.cc:288] T d55cc40d376e47b2ab531e102b8d28c0 P d02e909ab20f4a12b000cb5ca64212e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32317a2a19304a51a248edde90974bbb (127.21.83.1:44117), a4336684d7414e61b17d557ddde64653 (127.21.83.2:40429)
I20260807 07:02:26.555891 1412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:26.549947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:51813 (local address 127.21.83.2:40429)
0807 07:02:26.550156 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:26.550163 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:26.550943 (+ 780us) server_negotiation.cc:407] Connection header received
0807 07:02:26.551098 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:26.551102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:26.551163 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:26.551238 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:02:26.551811 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:26.552503 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:26.554216 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:26.555012 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:26.555124 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:26.555234 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:26.555487 (+ 253us) server_negotiation.cc:299] Negotiation successful
0807 07:02:26.555627 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 07:02:26.555994 1477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:26.549849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.2:40429 (local address 127.21.83.3:51813)
0807 07:02:26.550847 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:26.550862 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:26.550976 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:26.551275 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:26.551279 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:26.551305 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:02:26.551658 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:26.551667 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:26.552641 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:26.552646 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:26.554070 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:26.554083 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:26.555176 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:26.555182 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:26.555329 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:26.555354 (+ 25us) client_negotiation.cc:769] Sending connection context
0807 07:02:26.555443 (+ 89us) client_negotiation.cc:240] Negotiation successful
0807 07:02:26.555578 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":889,"mutex_wait_us":25,"spinlock_wait_cycles":23424,"thread_start_us":437,"threads_started":1}
I20260807 07:02:26.556208 1479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:26.550023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:43131 (local address 127.21.83.1:44117)
0807 07:02:26.551089 (+ 1066us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:26.551093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:26.551109 (+ 16us) server_negotiation.cc:407] Connection header received
0807 07:02:26.551169 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:26.551172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:26.551228 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:26.551323 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:02:26.552002 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:26.552706 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:26.554385 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:26.555257 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:26.555380 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:26.555506 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:26.555782 (+ 276us) server_negotiation.cc:299] Negotiation successful
0807 07:02:26.555920 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":946,"thread_start_us":839,"threads_started":1}
I20260807 07:02:26.556612 1478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:26.549809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:44117 (local address 127.21.83.3:43131)
0807 07:02:26.550577 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:26.550598 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:26.550759 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:26.551373 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:26.551379 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:26.551428 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:02:26.551840 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:26.551849 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:26.552848 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:26.552852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:26.554246 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:26.554258 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:26.555459 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:26.555465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:26.555639 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:26.555663 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 07:02:26.555739 (+ 76us) client_negotiation.cc:240] Negotiation successful
0807 07:02:26.555866 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"mutex_wait_us":56,"spinlock_wait_cycles":896,"thread_start_us":771,"threads_started":1}
I20260807 07:02:26.556610 1039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55cc40d376e47b2ab531e102b8d28c0" candidate_uuid: "d02e909ab20f4a12b000cb5ca64212e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4336684d7414e61b17d557ddde64653" is_pre_election: true
I20260807 07:02:26.556936 1039 raft_consensus.cc:2454] T d55cc40d376e47b2ab531e102b8d28c0 P a4336684d7414e61b17d557ddde64653 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d02e909ab20f4a12b000cb5ca64212e7 in term 0.
I20260807 07:02:26.557243 827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55cc40d376e47b2ab531e102b8d28c0" candidate_uuid: "d02e909ab20f4a12b000cb5ca64212e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32317a2a19304a51a248edde90974bbb" is_pre_election: true
I20260807 07:02:26.557425 1140 leader_election.cc:302] T d55cc40d376e47b2ab531e102b8d28c0 P d02e909ab20f4a12b000cb5ca64212e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4336684d7414e61b17d557ddde64653, d02e909ab20f4a12b000cb5ca64212e7; no voters:
I20260807 07:02:26.557505 827 raft_consensus.cc:2454] T d55cc40d376e47b2ab531e102b8d28c0 P 32317a2a19304a51a248edde90974bbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d02e909ab20f4a12b000cb5ca64212e7 in term 0.
I20260807 07:02:26.557824 1364 raft_consensus.cc:2781] T d55cc40d376e47b2ab531e102b8d28c0 P d02e909ab20f4a12b000cb5ca64212e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 07:02:26.557947 1364 raft_consensus.cc:482] T d55cc40d376e47b2ab531e102b8d28c0 P d02e909ab20f4a12b000cb5ca64212e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:02:26.558046 1364 raft_consensus.cc:3037] T d55cc40d376e47b2ab531e102b8d28c0 P d02e909ab20f4a12b000cb5ca64212e7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:02:26.560186 1364 raft_consensus.cc:504] T d55cc40d376e47b2ab531e102b8d28c0 P d02e909ab20f4a12b000cb5ca64212e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } } peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } }
I20260807 07:02:26.560834 1364 leader_election.cc:288] T d55cc40d376e47b2ab531e102b8d28c0 P d02e909ab20f4a12b000cb5ca64212e7 [CANDIDATE]: Term 1 election: Requested vote from peers 32317a2a19304a51a248edde90974bbb (127.21.83.1:44117), a4336684d7414e61b17d557ddde64653 (127.21.83.2:40429)
I20260807 07:02:26.561249 827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55cc40d376e47b2ab531e102b8d28c0" candidate_uuid: "d02e909ab20f4a12b000cb5ca64212e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32317a2a19304a51a248edde90974bbb"
I20260807 07:02:26.561339 1039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55cc40d376e47b2ab531e102b8d28c0" candidate_uuid: "d02e909ab20f4a12b000cb5ca64212e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4336684d7414e61b17d557ddde64653"
I20260807 07:02:26.561434 827 raft_consensus.cc:3037] T d55cc40d376e47b2ab531e102b8d28c0 P 32317a2a19304a51a248edde90974bbb [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:02:26.561570 1039 raft_consensus.cc:3037] T d55cc40d376e47b2ab531e102b8d28c0 P a4336684d7414e61b17d557ddde64653 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:02:26.563731 827 raft_consensus.cc:2454] T d55cc40d376e47b2ab531e102b8d28c0 P 32317a2a19304a51a248edde90974bbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d02e909ab20f4a12b000cb5ca64212e7 in term 1.
I20260807 07:02:26.563747 1039 raft_consensus.cc:2454] T d55cc40d376e47b2ab531e102b8d28c0 P a4336684d7414e61b17d557ddde64653 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d02e909ab20f4a12b000cb5ca64212e7 in term 1.
I20260807 07:02:26.564143 1143 leader_election.cc:302] T d55cc40d376e47b2ab531e102b8d28c0 P d02e909ab20f4a12b000cb5ca64212e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32317a2a19304a51a248edde90974bbb, d02e909ab20f4a12b000cb5ca64212e7; no voters:
I20260807 07:02:26.564466 1364 raft_consensus.cc:2781] T d55cc40d376e47b2ab531e102b8d28c0 P d02e909ab20f4a12b000cb5ca64212e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:02:26.564955 1364 raft_consensus.cc:686] T d55cc40d376e47b2ab531e102b8d28c0 P d02e909ab20f4a12b000cb5ca64212e7 [term 1 LEADER]: Becoming Leader. State: Replica: d02e909ab20f4a12b000cb5ca64212e7, State: Running, Role: LEADER
I20260807 07:02:26.565356 1364 consensus_queue.cc:237] T d55cc40d376e47b2ab531e102b8d28c0 P d02e909ab20f4a12b000cb5ca64212e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } } peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } }
I20260807 07:02:26.569561 604 catalog_manager.cc:5495] T d55cc40d376e47b2ab531e102b8d28c0 P d02e909ab20f4a12b000cb5ca64212e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d02e909ab20f4a12b000cb5ca64212e7 (127.21.83.3). New cstate: current_term: 1 leader_uuid: "d02e909ab20f4a12b000cb5ca64212e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } health_report { overall_health: HEALTHY } } }
I20260807 07:02:26.599388 1483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:26.593763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41558 (local address 127.21.83.3:35127)
0807 07:02:26.594328 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:26.594333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:26.594352 (+ 19us) server_negotiation.cc:407] Connection header received
0807 07:02:26.594439 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:26.594444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:26.594543 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:26.594718 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:02:26.595017 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:26.595828 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:26.597141 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:26.597614 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:26.597729 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:26.598109 (+ 380us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:02:26.598740 (+ 631us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:02:26.598893 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:26.599021 (+ 128us) server_negotiation.cc:299] Negotiation successful
0807 07:02:26.599104 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":305,"threads_started":1}
I20260807 07:02:26.603123 1039 raft_consensus.cc:1261] T d55cc40d376e47b2ab531e102b8d28c0 P a4336684d7414e61b17d557ddde64653 [term 1 FOLLOWER]: Refusing update from remote peer d02e909ab20f4a12b000cb5ca64212e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:02:26.603153 827 raft_consensus.cc:1261] T d55cc40d376e47b2ab531e102b8d28c0 P 32317a2a19304a51a248edde90974bbb [term 1 FOLLOWER]: Refusing update from remote peer d02e909ab20f4a12b000cb5ca64212e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:02:26.603785 1364 consensus_queue.cc:1035] T d55cc40d376e47b2ab531e102b8d28c0 P d02e909ab20f4a12b000cb5ca64212e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:02:26.604072 1480 consensus_queue.cc:1035] T d55cc40d376e47b2ab531e102b8d28c0 P d02e909ab20f4a12b000cb5ca64212e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:02:26.608497 1433 mvcc.cc:205] Tried to move back new op lower bound from 7315808856478679040 to 7315808856336572416. Current Snapshot: MvccSnapshot[applied={T|T < 7315808856478679040}]
I20260807 07:02:26.619225 600 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:02:26.619439 600 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:02:26.622397 600 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0935abe2905a450e8cbcfa9aead19321: Sending DeleteTablet for 3 replicas of tablet d55cc40d376e47b2ab531e102b8d28c0
I20260807 07:02:26.623190 799 tablet_service.cc:1514] Processing DeleteTablet for tablet d55cc40d376e47b2ab531e102b8d28c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:02:26 UTC) from {username='oryx', principal='oryx/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:58544
I20260807 07:02:26.623332 1001 tablet_service.cc:1514] Processing DeleteTablet for tablet d55cc40d376e47b2ab531e102b8d28c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:02:26 UTC) from {username='oryx', principal='oryx/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:41708
I20260807 07:02:26.623479 1449 tablet_replica.cc:330] T d55cc40d376e47b2ab531e102b8d28c0 P 32317a2a19304a51a248edde90974bbb: stopping tablet replica
I20260807 07:02:26.623582 1451 tablet_replica.cc:330] T d55cc40d376e47b2ab531e102b8d28c0 P a4336684d7414e61b17d557ddde64653: stopping tablet replica
I20260807 07:02:26.623682 1449 raft_consensus.cc:2229] T d55cc40d376e47b2ab531e102b8d28c0 P 32317a2a19304a51a248edde90974bbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:02:26.623764 1451 raft_consensus.cc:2229] T d55cc40d376e47b2ab531e102b8d28c0 P a4336684d7414e61b17d557ddde64653 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:02:26.623919 1449 raft_consensus.cc:2258] T d55cc40d376e47b2ab531e102b8d28c0 P 32317a2a19304a51a248edde90974bbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:02:26.623978 1451 raft_consensus.cc:2258] T d55cc40d376e47b2ab531e102b8d28c0 P a4336684d7414e61b17d557ddde64653 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:02:26.624781 1217 tablet_service.cc:1514] Processing DeleteTablet for tablet d55cc40d376e47b2ab531e102b8d28c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:02:26 UTC) from {username='oryx', principal='oryx/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:41542
I20260807 07:02:26.625238 1449 ts_tablet_manager.cc:1905] T d55cc40d376e47b2ab531e102b8d28c0 P 32317a2a19304a51a248edde90974bbb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:02:26.625054 600 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49796:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260807 07:02:26.625335 1451 ts_tablet_manager.cc:1905] T d55cc40d376e47b2ab531e102b8d28c0 P a4336684d7414e61b17d557ddde64653: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260807 07:02:26.625727 600 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 07:02:26.629051 1450 tablet_replica.cc:330] T d55cc40d376e47b2ab531e102b8d28c0 P d02e909ab20f4a12b000cb5ca64212e7: stopping tablet replica
I20260807 07:02:26.629273 1450 raft_consensus.cc:2229] T d55cc40d376e47b2ab531e102b8d28c0 P d02e909ab20f4a12b000cb5ca64212e7 [term 1 LEADER]: Raft consensus shutting down.
I20260807 07:02:26.629760 1450 raft_consensus.cc:2258] T d55cc40d376e47b2ab531e102b8d28c0 P d02e909ab20f4a12b000cb5ca64212e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:02:26.630957 1450 ts_tablet_manager.cc:1905] T d55cc40d376e47b2ab531e102b8d28c0 P d02e909ab20f4a12b000cb5ca64212e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:02:26.632234 1449 ts_tablet_manager.cc:1918] T d55cc40d376e47b2ab531e102b8d28c0 P 32317a2a19304a51a248edde90974bbb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:02:26.632416 1449 log.cc:1196] T d55cc40d376e47b2ab531e102b8d28c0 P 32317a2a19304a51a248edde90974bbb: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-0/wal/wals/d55cc40d376e47b2ab531e102b8d28c0
I20260807 07:02:26.632859 1449 ts_tablet_manager.cc:1939] T d55cc40d376e47b2ab531e102b8d28c0 P 32317a2a19304a51a248edde90974bbb: Deleting consensus metadata
I20260807 07:02:26.633812 1451 ts_tablet_manager.cc:1918] T d55cc40d376e47b2ab531e102b8d28c0 P a4336684d7414e61b17d557ddde64653: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:02:26.634007 1451 log.cc:1196] T d55cc40d376e47b2ab531e102b8d28c0 P a4336684d7414e61b17d557ddde64653: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-1/wal/wals/d55cc40d376e47b2ab531e102b8d28c0
I20260807 07:02:26.633981 588 catalog_manager.cc:4841] TS 32317a2a19304a51a248edde90974bbb (127.21.83.1:44117): tablet d55cc40d376e47b2ab531e102b8d28c0 (table test-table [id=f2e26e0c98bf4d7f891a5bdd1b86b3fc]) successfully deleted
I20260807 07:02:26.634467 1451 ts_tablet_manager.cc:1939] T d55cc40d376e47b2ab531e102b8d28c0 P a4336684d7414e61b17d557ddde64653: Deleting consensus metadata
I20260807 07:02:26.635609 585 catalog_manager.cc:4841] TS a4336684d7414e61b17d557ddde64653 (127.21.83.2:40429): tablet d55cc40d376e47b2ab531e102b8d28c0 (table test-table [id=f2e26e0c98bf4d7f891a5bdd1b86b3fc]) successfully deleted
I20260807 07:02:26.637074 1450 ts_tablet_manager.cc:1918] T d55cc40d376e47b2ab531e102b8d28c0 P d02e909ab20f4a12b000cb5ca64212e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:02:26.637243 1450 log.cc:1196] T d55cc40d376e47b2ab531e102b8d28c0 P d02e909ab20f4a12b000cb5ca64212e7: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-2/wal/wals/d55cc40d376e47b2ab531e102b8d28c0
I20260807 07:02:26.637696 1450 ts_tablet_manager.cc:1939] T d55cc40d376e47b2ab531e102b8d28c0 P d02e909ab20f4a12b000cb5ca64212e7: Deleting consensus metadata
I20260807 07:02:26.638684 585 catalog_manager.cc:4841] TS d02e909ab20f4a12b000cb5ca64212e7 (127.21.83.3:35127): tablet d55cc40d376e47b2ab531e102b8d28c0 (table test-table [id=f2e26e0c98bf4d7f891a5bdd1b86b3fc]) successfully deleted
I20260807 07:02:26.640303 799 tablet_service.cc:1467] Processing CreateTablet for tablet 7aa76b2a2ae749719af8d34ff5b89de3 (DEFAULT_TABLE table=test-table [id=4fda65796bb94b348003b3c9b64b7eb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:02:26.640740 799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7aa76b2a2ae749719af8d34ff5b89de3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:02:26.640774 1001 tablet_service.cc:1467] Processing CreateTablet for tablet 7aa76b2a2ae749719af8d34ff5b89de3 (DEFAULT_TABLE table=test-table [id=4fda65796bb94b348003b3c9b64b7eb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:02:26.641012 1217 tablet_service.cc:1467] Processing CreateTablet for tablet 7aa76b2a2ae749719af8d34ff5b89de3 (DEFAULT_TABLE table=test-table [id=4fda65796bb94b348003b3c9b64b7eb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:02:26.641240 1001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7aa76b2a2ae749719af8d34ff5b89de3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:02:26.641423 1217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7aa76b2a2ae749719af8d34ff5b89de3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:02:26.647544 1471 tablet_bootstrap.cc:492] T 7aa76b2a2ae749719af8d34ff5b89de3 P 32317a2a19304a51a248edde90974bbb: Bootstrap starting.
I20260807 07:02:26.648353 1470 tablet_bootstrap.cc:492] T 7aa76b2a2ae749719af8d34ff5b89de3 P d02e909ab20f4a12b000cb5ca64212e7: Bootstrap starting.
I20260807 07:02:26.648710 1472 tablet_bootstrap.cc:492] T 7aa76b2a2ae749719af8d34ff5b89de3 P a4336684d7414e61b17d557ddde64653: Bootstrap starting.
I20260807 07:02:26.649374 1471 tablet_bootstrap.cc:654] T 7aa76b2a2ae749719af8d34ff5b89de3 P 32317a2a19304a51a248edde90974bbb: Neither blocks nor log segments found. Creating new log.
I20260807 07:02:26.650050 1470 tablet_bootstrap.cc:654] T 7aa76b2a2ae749719af8d34ff5b89de3 P d02e909ab20f4a12b000cb5ca64212e7: Neither blocks nor log segments found. Creating new log.
I20260807 07:02:26.650322 1472 tablet_bootstrap.cc:654] T 7aa76b2a2ae749719af8d34ff5b89de3 P a4336684d7414e61b17d557ddde64653: Neither blocks nor log segments found. Creating new log.
I20260807 07:02:26.660231 1470 tablet_bootstrap.cc:492] T 7aa76b2a2ae749719af8d34ff5b89de3 P d02e909ab20f4a12b000cb5ca64212e7: No bootstrap required, opened a new log
I20260807 07:02:26.660420 1470 ts_tablet_manager.cc:1397] T 7aa76b2a2ae749719af8d34ff5b89de3 P d02e909ab20f4a12b000cb5ca64212e7: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20260807 07:02:26.660557 1472 tablet_bootstrap.cc:492] T 7aa76b2a2ae749719af8d34ff5b89de3 P a4336684d7414e61b17d557ddde64653: No bootstrap required, opened a new log
I20260807 07:02:26.660773 1472 ts_tablet_manager.cc:1397] T 7aa76b2a2ae749719af8d34ff5b89de3 P a4336684d7414e61b17d557ddde64653: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.005s
I20260807 07:02:26.661356 1470 raft_consensus.cc:348] T 7aa76b2a2ae749719af8d34ff5b89de3 P d02e909ab20f4a12b000cb5ca64212e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } } peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } }
I20260807 07:02:26.661701 1470 raft_consensus.cc:374] T 7aa76b2a2ae749719af8d34ff5b89de3 P d02e909ab20f4a12b000cb5ca64212e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:02:26.661604 1472 raft_consensus.cc:348] T 7aa76b2a2ae749719af8d34ff5b89de3 P a4336684d7414e61b17d557ddde64653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } } peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } }
I20260807 07:02:26.661788 1470 raft_consensus.cc:729] T 7aa76b2a2ae749719af8d34ff5b89de3 P d02e909ab20f4a12b000cb5ca64212e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d02e909ab20f4a12b000cb5ca64212e7, State: Initialized, Role: FOLLOWER
I20260807 07:02:26.661912 1472 raft_consensus.cc:374] T 7aa76b2a2ae749719af8d34ff5b89de3 P a4336684d7414e61b17d557ddde64653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:02:26.662007 1472 raft_consensus.cc:729] T 7aa76b2a2ae749719af8d34ff5b89de3 P a4336684d7414e61b17d557ddde64653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4336684d7414e61b17d557ddde64653, State: Initialized, Role: FOLLOWER
I20260807 07:02:26.662096 1470 consensus_queue.cc:260] T 7aa76b2a2ae749719af8d34ff5b89de3 P d02e909ab20f4a12b000cb5ca64212e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } } peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } }
I20260807 07:02:26.662287 1472 consensus_queue.cc:260] T 7aa76b2a2ae749719af8d34ff5b89de3 P a4336684d7414e61b17d557ddde64653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } } peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } }
I20260807 07:02:26.662675 1470 ts_tablet_manager.cc:1428] T 7aa76b2a2ae749719af8d34ff5b89de3 P d02e909ab20f4a12b000cb5ca64212e7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260807 07:02:26.664083 1471 tablet_bootstrap.cc:492] T 7aa76b2a2ae749719af8d34ff5b89de3 P 32317a2a19304a51a248edde90974bbb: No bootstrap required, opened a new log
I20260807 07:02:26.664253 1471 ts_tablet_manager.cc:1397] T 7aa76b2a2ae749719af8d34ff5b89de3 P 32317a2a19304a51a248edde90974bbb: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.007s
I20260807 07:02:26.665035 1471 raft_consensus.cc:348] T 7aa76b2a2ae749719af8d34ff5b89de3 P 32317a2a19304a51a248edde90974bbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } } peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } }
I20260807 07:02:26.665295 1471 raft_consensus.cc:374] T 7aa76b2a2ae749719af8d34ff5b89de3 P 32317a2a19304a51a248edde90974bbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:02:26.665385 1471 raft_consensus.cc:729] T 7aa76b2a2ae749719af8d34ff5b89de3 P 32317a2a19304a51a248edde90974bbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32317a2a19304a51a248edde90974bbb, State: Initialized, Role: FOLLOWER
I20260807 07:02:26.665619 1471 consensus_queue.cc:260] T 7aa76b2a2ae749719af8d34ff5b89de3 P 32317a2a19304a51a248edde90974bbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } } peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } }
I20260807 07:02:26.666178 1471 ts_tablet_manager.cc:1428] T 7aa76b2a2ae749719af8d34ff5b89de3 P 32317a2a19304a51a248edde90974bbb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260807 07:02:26.666775 1472 ts_tablet_manager.cc:1428] T 7aa76b2a2ae749719af8d34ff5b89de3 P a4336684d7414e61b17d557ddde64653: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260807 07:02:27.025547 1365 raft_consensus.cc:482] T 7aa76b2a2ae749719af8d34ff5b89de3 P 32317a2a19304a51a248edde90974bbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:02:27.025799 1365 raft_consensus.cc:504] T 7aa76b2a2ae749719af8d34ff5b89de3 P 32317a2a19304a51a248edde90974bbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } } peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } }
I20260807 07:02:27.026587 1365 leader_election.cc:288] T 7aa76b2a2ae749719af8d34ff5b89de3 P 32317a2a19304a51a248edde90974bbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4336684d7414e61b17d557ddde64653 (127.21.83.2:40429), d02e909ab20f4a12b000cb5ca64212e7 (127.21.83.3:35127)
I20260807 07:02:27.027226 1039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aa76b2a2ae749719af8d34ff5b89de3" candidate_uuid: "32317a2a19304a51a248edde90974bbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4336684d7414e61b17d557ddde64653" is_pre_election: true
I20260807 07:02:27.027232 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aa76b2a2ae749719af8d34ff5b89de3" candidate_uuid: "32317a2a19304a51a248edde90974bbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d02e909ab20f4a12b000cb5ca64212e7" is_pre_election: true
I20260807 07:02:27.027518 1039 raft_consensus.cc:2454] T 7aa76b2a2ae749719af8d34ff5b89de3 P a4336684d7414e61b17d557ddde64653 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32317a2a19304a51a248edde90974bbb in term 0.
I20260807 07:02:27.027536 1241 raft_consensus.cc:2454] T 7aa76b2a2ae749719af8d34ff5b89de3 P d02e909ab20f4a12b000cb5ca64212e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32317a2a19304a51a248edde90974bbb in term 0.
I20260807 07:02:27.027998 727 leader_election.cc:302] T 7aa76b2a2ae749719af8d34ff5b89de3 P 32317a2a19304a51a248edde90974bbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32317a2a19304a51a248edde90974bbb, a4336684d7414e61b17d557ddde64653; no voters:
I20260807 07:02:27.028257 1365 raft_consensus.cc:2781] T 7aa76b2a2ae749719af8d34ff5b89de3 P 32317a2a19304a51a248edde90974bbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 07:02:27.028389 1365 raft_consensus.cc:482] T 7aa76b2a2ae749719af8d34ff5b89de3 P 32317a2a19304a51a248edde90974bbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:02:27.028486 1365 raft_consensus.cc:3037] T 7aa76b2a2ae749719af8d34ff5b89de3 P 32317a2a19304a51a248edde90974bbb [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:02:27.030867 1365 raft_consensus.cc:504] T 7aa76b2a2ae749719af8d34ff5b89de3 P 32317a2a19304a51a248edde90974bbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } } peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } }
I20260807 07:02:27.031443 1365 leader_election.cc:288] T 7aa76b2a2ae749719af8d34ff5b89de3 P 32317a2a19304a51a248edde90974bbb [CANDIDATE]: Term 1 election: Requested vote from peers a4336684d7414e61b17d557ddde64653 (127.21.83.2:40429), d02e909ab20f4a12b000cb5ca64212e7 (127.21.83.3:35127)
I20260807 07:02:27.031876 1039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aa76b2a2ae749719af8d34ff5b89de3" candidate_uuid: "32317a2a19304a51a248edde90974bbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4336684d7414e61b17d557ddde64653"
I20260807 07:02:27.031980 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aa76b2a2ae749719af8d34ff5b89de3" candidate_uuid: "32317a2a19304a51a248edde90974bbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d02e909ab20f4a12b000cb5ca64212e7"
I20260807 07:02:27.032078 1039 raft_consensus.cc:3037] T 7aa76b2a2ae749719af8d34ff5b89de3 P a4336684d7414e61b17d557ddde64653 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:02:27.032231 1241 raft_consensus.cc:3037] T 7aa76b2a2ae749719af8d34ff5b89de3 P d02e909ab20f4a12b000cb5ca64212e7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:02:27.034152 1039 raft_consensus.cc:2454] T 7aa76b2a2ae749719af8d34ff5b89de3 P a4336684d7414e61b17d557ddde64653 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32317a2a19304a51a248edde90974bbb in term 1.
I20260807 07:02:27.034575 727 leader_election.cc:302] T 7aa76b2a2ae749719af8d34ff5b89de3 P 32317a2a19304a51a248edde90974bbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32317a2a19304a51a248edde90974bbb, a4336684d7414e61b17d557ddde64653; no voters:
I20260807 07:02:27.034842 1365 raft_consensus.cc:2781] T 7aa76b2a2ae749719af8d34ff5b89de3 P 32317a2a19304a51a248edde90974bbb [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:02:27.035275 1365 raft_consensus.cc:686] T 7aa76b2a2ae749719af8d34ff5b89de3 P 32317a2a19304a51a248edde90974bbb [term 1 LEADER]: Becoming Leader. State: Replica: 32317a2a19304a51a248edde90974bbb, State: Running, Role: LEADER
I20260807 07:02:27.035514 1241 raft_consensus.cc:2454] T 7aa76b2a2ae749719af8d34ff5b89de3 P d02e909ab20f4a12b000cb5ca64212e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32317a2a19304a51a248edde90974bbb in term 1.
I20260807 07:02:27.036000 1365 consensus_queue.cc:237] T 7aa76b2a2ae749719af8d34ff5b89de3 P 32317a2a19304a51a248edde90974bbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } } peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } }
I20260807 07:02:27.039772 600 catalog_manager.cc:5495] T 7aa76b2a2ae749719af8d34ff5b89de3 P 32317a2a19304a51a248edde90974bbb reported cstate change: term changed from 0 to 1, leader changed from <none> to 32317a2a19304a51a248edde90974bbb (127.21.83.1). New cstate: current_term: 1 leader_uuid: "32317a2a19304a51a248edde90974bbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32317a2a19304a51a248edde90974bbb" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 44117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 } health_report { overall_health: UNKNOWN } } }
I20260807 07:02:27.136158 1507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:27.126450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58552 (local address 127.21.83.1:44117)
0807 07:02:27.126935 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:27.126941 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:27.130280 (+ 3339us) server_negotiation.cc:407] Connection header received
0807 07:02:27.130477 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:27.130482 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:27.130610 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:27.130794 (+ 184us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:02:27.131156 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:27.132213 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:27.133642 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:27.134275 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:27.134409 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:27.134826 (+ 417us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:02:27.135478 (+ 652us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:02:27.135616 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:27.135730 (+ 114us) server_negotiation.cc:299] Negotiation successful
0807 07:02:27.135814 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":250,"threads_started":1}
I20260807 07:02:27.151575 1514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:27.146008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:55265 (local address 127.21.83.2:40429)
0807 07:02:27.146924 (+ 916us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:27.146931 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:27.146950 (+ 19us) server_negotiation.cc:407] Connection header received
0807 07:02:27.147030 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:27.147035 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:27.147105 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:27.147187 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:02:27.147698 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:27.148435 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:27.149829 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:27.150264 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:27.150377 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:27.150824 (+ 447us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:02:27.150929 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:02:27.151007 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:27.151152 (+ 145us) server_negotiation.cc:299] Negotiation successful
0807 07:02:27.151236 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":807,"thread_start_us":660,"threads_started":1}
I20260807 07:02:27.151639 1513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:27.145963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.2:40429 (local address 127.21.83.1:55265)
0807 07:02:27.146764 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:27.146785 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:27.146922 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:27.147223 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:27.147227 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:27.147239 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 07:02:27.147528 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:27.147538 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:27.148560 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:27.148564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:27.149690 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:27.149703 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:27.150439 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:27.150445 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:27.150577 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:27.150644 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 07:02:27.151025 (+ 381us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 07:02:27.151037 (+ 12us) client_negotiation.cc:769] Sending connection context
0807 07:02:27.151107 (+ 70us) client_negotiation.cc:240] Negotiation successful
0807 07:02:27.151267 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":374,"threads_started":1}
I20260807 07:02:27.160095 1241 raft_consensus.cc:1261] T 7aa76b2a2ae749719af8d34ff5b89de3 P d02e909ab20f4a12b000cb5ca64212e7 [term 1 FOLLOWER]: Refusing update from remote peer 32317a2a19304a51a248edde90974bbb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:02:27.160153 1039 raft_consensus.cc:1261] T 7aa76b2a2ae749719af8d34ff5b89de3 P a4336684d7414e61b17d557ddde64653 [term 1 FOLLOWER]: Refusing update from remote peer 32317a2a19304a51a248edde90974bbb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:02:27.160792 1492 consensus_queue.cc:1035] T 7aa76b2a2ae749719af8d34ff5b89de3 P 32317a2a19304a51a248edde90974bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4336684d7414e61b17d557ddde64653" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 40429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:02:27.161203 1365 consensus_queue.cc:1035] T 7aa76b2a2ae749719af8d34ff5b89de3 P 32317a2a19304a51a248edde90974bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d02e909ab20f4a12b000cb5ca64212e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 35127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:02:27.194515 1507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:27.188630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:46123 (local address 127.21.83.1:44117)
0807 07:02:27.188806 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:27.188812 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:27.189350 (+ 538us) server_negotiation.cc:407] Connection header received
0807 07:02:27.189634 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:27.189640 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:27.189732 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:27.189824 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:02:27.190220 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:27.190936 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:27.192643 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:27.193179 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:27.193309 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:27.193853 (+ 544us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:02:27.194017 (+ 164us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:02:27.194092 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:27.194212 (+ 120us) server_negotiation.cc:299] Negotiation successful
0807 07:02:27.194287 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260807 07:02:27.194702 1522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:27.188540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:44117 (local address 127.21.83.2:46123)
0807 07:02:27.189254 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:27.189275 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:27.189462 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:27.189871 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:27.189874 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:27.189886 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 07:02:27.190096 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:27.190105 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:27.191152 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:27.191156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:27.192452 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:27.192470 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:27.193342 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:27.193348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:27.193550 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:27.193685 (+ 135us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 07:02:27.194102 (+ 417us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 07:02:27.194113 (+ 11us) client_negotiation.cc:769] Sending connection context
0807 07:02:27.194180 (+ 67us) client_negotiation.cc:240] Negotiation successful
0807 07:02:27.194301 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":372,"threads_started":1}
I20260807 07:02:27.230875 600 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:02:27.231161 600 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:02:27.234520 600 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0935abe2905a450e8cbcfa9aead19321: Sending DeleteTablet for 3 replicas of tablet 7aa76b2a2ae749719af8d34ff5b89de3
I20260807 07:02:27.235363 799 tablet_service.cc:1514] Processing DeleteTablet for tablet 7aa76b2a2ae749719af8d34ff5b89de3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:02:27 UTC) from {username='oryx', principal='oryx/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:58544
I20260807 07:02:27.235476 1217 tablet_service.cc:1514] Processing DeleteTablet for tablet 7aa76b2a2ae749719af8d34ff5b89de3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:02:27 UTC) from {username='oryx', principal='oryx/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:41542
I20260807 07:02:27.236104 1525 tablet_replica.cc:330] T 7aa76b2a2ae749719af8d34ff5b89de3 P 32317a2a19304a51a248edde90974bbb: stopping tablet replica
I20260807 07:02:27.236197 1001 tablet_service.cc:1514] Processing DeleteTablet for tablet 7aa76b2a2ae749719af8d34ff5b89de3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:02:27 UTC) from {username='oryx', principal='oryx/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:41708
I20260807 07:02:27.236455 1525 raft_consensus.cc:2229] T 7aa76b2a2ae749719af8d34ff5b89de3 P 32317a2a19304a51a248edde90974bbb [term 1 LEADER]: Raft consensus shutting down.
I20260807 07:02:27.237167 1525 raft_consensus.cc:2258] T 7aa76b2a2ae749719af8d34ff5b89de3 P 32317a2a19304a51a248edde90974bbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:02:27.238381 1525 ts_tablet_manager.cc:1905] T 7aa76b2a2ae749719af8d34ff5b89de3 P 32317a2a19304a51a248edde90974bbb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:02:27.239835 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 704
I20260807 07:02:27.246802 1525 ts_tablet_manager.cc:1918] T 7aa76b2a2ae749719af8d34ff5b89de3 P 32317a2a19304a51a248edde90974bbb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 07:02:27.247140 1525 log.cc:1196] T 7aa76b2a2ae749719af8d34ff5b89de3 P 32317a2a19304a51a248edde90974bbb: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-0/wal/wals/7aa76b2a2ae749719af8d34ff5b89de3
I20260807 07:02:27.247740 1525 ts_tablet_manager.cc:1939] T 7aa76b2a2ae749719af8d34ff5b89de3 P 32317a2a19304a51a248edde90974bbb: Deleting consensus metadata
I20260807 07:02:27.249586 588 catalog_manager.cc:4841] TS 32317a2a19304a51a248edde90974bbb (127.21.83.1:44117): tablet 7aa76b2a2ae749719af8d34ff5b89de3 (table test-table [id=4fda65796bb94b348003b3c9b64b7eb7]) successfully deleted
I20260807 07:02:27.251875 1527 tablet_replica.cc:330] T 7aa76b2a2ae749719af8d34ff5b89de3 P a4336684d7414e61b17d557ddde64653: stopping tablet replica
I20260807 07:02:27.252112 1527 raft_consensus.cc:2229] T 7aa76b2a2ae749719af8d34ff5b89de3 P a4336684d7414e61b17d557ddde64653 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:02:27.252388 1527 raft_consensus.cc:2258] T 7aa76b2a2ae749719af8d34ff5b89de3 P a4336684d7414e61b17d557ddde64653 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:02:27.253592 1527 ts_tablet_manager.cc:1905] T 7aa76b2a2ae749719af8d34ff5b89de3 P a4336684d7414e61b17d557ddde64653: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:02:27.256765 1526 tablet_replica.cc:330] T 7aa76b2a2ae749719af8d34ff5b89de3 P d02e909ab20f4a12b000cb5ca64212e7: stopping tablet replica
I20260807 07:02:27.256978 1526 raft_consensus.cc:2229] T 7aa76b2a2ae749719af8d34ff5b89de3 P d02e909ab20f4a12b000cb5ca64212e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:02:27.257207 1526 raft_consensus.cc:2258] T 7aa76b2a2ae749719af8d34ff5b89de3 P d02e909ab20f4a12b000cb5ca64212e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:02:27.258248 1526 ts_tablet_manager.cc:1905] T 7aa76b2a2ae749719af8d34ff5b89de3 P d02e909ab20f4a12b000cb5ca64212e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:02:27.262092 1527 ts_tablet_manager.cc:1918] T 7aa76b2a2ae749719af8d34ff5b89de3 P a4336684d7414e61b17d557ddde64653: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 07:02:27.262266 1527 log.cc:1196] T 7aa76b2a2ae749719af8d34ff5b89de3 P a4336684d7414e61b17d557ddde64653: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-1/wal/wals/7aa76b2a2ae749719af8d34ff5b89de3
I20260807 07:02:27.262753 1527 ts_tablet_manager.cc:1939] T 7aa76b2a2ae749719af8d34ff5b89de3 P a4336684d7414e61b17d557ddde64653: Deleting consensus metadata
I20260807 07:02:27.263906 585 catalog_manager.cc:4841] TS a4336684d7414e61b17d557ddde64653 (127.21.83.2:40429): tablet 7aa76b2a2ae749719af8d34ff5b89de3 (table test-table [id=4fda65796bb94b348003b3c9b64b7eb7]) successfully deleted
I20260807 07:02:27.265277 1526 ts_tablet_manager.cc:1918] T 7aa76b2a2ae749719af8d34ff5b89de3 P d02e909ab20f4a12b000cb5ca64212e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 07:02:27.265404 1526 log.cc:1196] T 7aa76b2a2ae749719af8d34ff5b89de3 P d02e909ab20f4a12b000cb5ca64212e7: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786086078080656-21836-0/minicluster-data/ts-2/wal/wals/7aa76b2a2ae749719af8d34ff5b89de3
I20260807 07:02:27.265796 1526 ts_tablet_manager.cc:1939] T 7aa76b2a2ae749719af8d34ff5b89de3 P d02e909ab20f4a12b000cb5ca64212e7: Deleting consensus metadata
I20260807 07:02:27.266672 585 catalog_manager.cc:4841] TS d02e909ab20f4a12b000cb5ca64212e7 (127.21.83.3:35127): tablet 7aa76b2a2ae749719af8d34ff5b89de3 (table test-table [id=4fda65796bb94b348003b3c9b64b7eb7]) successfully deleted
Aug 07 07:02:27 dist-test-slave-ncp9 krb5kdc[544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.83.1@KRBTEST.COM
I20260807 07:02:27.275697 1507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:27.241245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58566 (local address 127.21.83.1:44117)
0807 07:02:27.241433 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:27.241439 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:27.256020 (+ 14581us) server_negotiation.cc:407] Connection header received
0807 07:02:27.256256 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:27.256262 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:27.256352 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:27.256468 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:27.258521 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:27.259767 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:27.262178 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:27.262904 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:27.263032 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:27.268208 (+ 5176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:27.268230 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:27.268252 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:27.268299 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:27.272454 (+ 4155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:27.273307 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:27.273317 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:27.273321 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:27.273524 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:27.273909 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:27.273915 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:27.273918 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:27.274410 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:27.274758 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:27.275007 (+ 249us) server_negotiation.cc:299] Negotiation successful
0807 07:02:27.275255 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260807 07:02:27.276492 898 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:02:27.389063 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 704
I20260807 07:02:27.421221 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 912
Aug 07 07:02:27 dist-test-slave-ncp9 krb5kdc[544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.83.2@KRBTEST.COM
I20260807 07:02:27.435381 1514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:27.422166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41730 (local address 127.21.83.2:40429)
0807 07:02:27.422347 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:27.422352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:27.422369 (+ 17us) server_negotiation.cc:407] Connection header received
0807 07:02:27.422448 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:27.422451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:27.422512 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:27.422589 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:27.423727 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:27.424482 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:27.425781 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:27.426162 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:27.426266 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:27.430047 (+ 3781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:27.430060 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:27.430075 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:27.430111 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:27.432823 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:27.433532 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:27.433539 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:27.433541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:27.433667 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:27.434024 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:27.434028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:27.434031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:27.434488 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:27.434728 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:27.434944 (+ 216us) server_negotiation.cc:299] Negotiation successful
0807 07:02:27.435126 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 07:02:27.436018 1102 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:02:27.588975 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 912
I20260807 07:02:27.628295 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 1115
Aug 07 07:02:27 dist-test-slave-ncp9 krb5kdc[544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.83.3@KRBTEST.COM
I20260807 07:02:27.644565 1536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:27.629147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41564 (local address 127.21.83.3:35127)
0807 07:02:27.629668 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:27.629673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:27.629692 (+ 19us) server_negotiation.cc:407] Connection header received
0807 07:02:27.629749 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:27.629753 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:27.629816 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:27.629909 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:27.631325 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:27.632083 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:27.633349 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:27.633890 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:27.634020 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:27.638407 (+ 4387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:27.638427 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:27.638443 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:27.638487 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:27.641599 (+ 3112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:27.642406 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:27.642414 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:27.642416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:27.642563 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:27.642954 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:27.642958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:27.642960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:27.643348 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:27.643654 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:27.643966 (+ 312us) server_negotiation.cc:299] Negotiation successful
0807 07:02:27.644234 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":278,"threads_started":1}
I20260807 07:02:27.645428 1316 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:02:27.772495 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 1115
I20260807 07:02:27.804765 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 562
I20260807 07:02:27.806051 663 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:02:27.926959 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 562
2026-08-07T07:02:27Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6970 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786086078080656-21836-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 07:02:28 dist-test-slave-ncp9 krb5kdc[1542](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 07:02:28 dist-test-slave-ncp9 krb5kdc[1542](info): set up 2 sockets
Aug 07 07:02:28 dist-test-slave-ncp9 krb5kdc[1542](info): commencing operation
krb5kdc: starting...
W20260807 07:02:29.137401 21836 mini_kdc.cc:121] Time spent starting KDC: real 1.161s user 0.003s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Aug 07 07:02:29 dist-test-slave-ncp9 krb5kdc[1542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T07:02:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:02:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
I20260807 07:02:29.423738 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:35211
--webserver_interface=127.21.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:35231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:35211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:02:29.764967 1560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:02:29.765254 1560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:02:29.765383 1560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:02:29.772328 1560 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:02:29.772431 1560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:02:29.772511 1560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:02:29.772595 1560 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:02:29.772660 1560 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:02:29.782141 1560 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:35231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:35211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:35211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:02:29.783322 1560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:02:29.784538 1560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:02:29.795387 1569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:29.795590 1571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:29.795591 1567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:02:29.796416 1560 server_base.cc:1034] running on GCE node
I20260807 07:02:29.797139 1560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:02:29.798424 1560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:02:29.799609 1560 hybrid_clock.cc:648] HybridClock initialized: now 1786086149799561 us; error 43 us; skew 500 ppm
I20260807 07:02:29.799911 1560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-07T07:02:29Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1945 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 07:02:29 dist-test-slave-ncp9 krb5kdc[1581](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 07:02:29 dist-test-slave-ncp9 krb5kdc[1581](info): set up 2 sockets
Aug 07 07:02:29 dist-test-slave-ncp9 krb5kdc[1581](info): commencing operation
krb5kdc: starting...
W20260807 07:02:31.073789 21836 mini_kdc.cc:121] Time spent starting KDC: real 1.152s user 0.001s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Aug 07 07:02:31 dist-test-slave-ncp9 krb5kdc[1581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T07:02:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:02:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
I20260807 07:02:31.326289 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:38779
--webserver_interface=127.21.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:34761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:38779
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:02:31.654170 1600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:02:31.654443 1600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:02:31.654573 1600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:02:31.661619 1600 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:02:31.661716 1600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:02:31.661799 1600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:02:31.661880 1600 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:02:31.661942 1600 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:02:31.671597 1600 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:34761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:38779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:38779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:02:31.673090 1600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:02:31.674443 1600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:02:31.686599 1616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:31.686595 1611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:31.686887 1609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:02:31.687417 1600 server_base.cc:1034] running on GCE node
I20260807 07:02:31.688256 1600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:02:31.689899 1600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:02:31.691124 1600 hybrid_clock.cc:648] HybridClock initialized: now 1786086151691062 us; error 64 us; skew 500 ppm
I20260807 07:02:31.691511 1600 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:02:31 dist-test-slave-ncp9 krb5kdc[1581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086151, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:02:31.694746 1600 init.cc:377] Logged in from keytab as kudu/127.21.83.62@KRBTEST.COM (short username kudu)
I20260807 07:02:31.696347 1600 webserver.cc:460] Webserver started at http://127.21.83.62:45487/ using document root <none> and password file <none>
I20260807 07:02:31.697108 1600 fs_manager.cc:359] Metadata directory not provided
I20260807 07:02:31.697232 1600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:02:31.697530 1600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:02:31.700587 1600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "73652b2d3c38462db13ad7e166a07937"
format_stamp: "Formatted at 2026-08-07 07:02:31 on dist-test-slave-ncp9"
server_key: "13a463efe0e88c2f993fdd1814bddc1b"
server_key_iv: "abb8062dbcea04a3abec6f7003cf1ddc"
server_key_version: "encryptionkey@0"
I20260807 07:02:31.701364 1600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "73652b2d3c38462db13ad7e166a07937"
format_stamp: "Formatted at 2026-08-07 07:02:31 on dist-test-slave-ncp9"
server_key: "13a463efe0e88c2f993fdd1814bddc1b"
server_key_iv: "abb8062dbcea04a3abec6f7003cf1ddc"
server_key_version: "encryptionkey@0"
I20260807 07:02:31.707212 1600 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260807 07:02:31.710487 1627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:31.711690 1600 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 07:02:31.711841 1600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "73652b2d3c38462db13ad7e166a07937"
format_stamp: "Formatted at 2026-08-07 07:02:31 on dist-test-slave-ncp9"
server_key: "13a463efe0e88c2f993fdd1814bddc1b"
server_key_iv: "abb8062dbcea04a3abec6f7003cf1ddc"
server_key_version: "encryptionkey@0"
I20260807 07:02:31.712013 1600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:02:31.736764 1600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:02:31.741127 1600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:02:31.741441 1600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:02:31.760249 1600 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.62:38779
I20260807 07:02:31.760272 1699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.62:38779 every 8 connection(s)
I20260807 07:02:31.761750 1600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:02:31.765808 1700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:02:31.770342 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 1600
I20260807 07:02:31.770587 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:02:31.771283 21836 external_mini_cluster.cc:1428] Setting key 398e49c5cac2a605b315f7323e97f631
I20260807 07:02:31.778103 1700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73652b2d3c38462db13ad7e166a07937: Bootstrap starting.
I20260807 07:02:31.781749 1700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73652b2d3c38462db13ad7e166a07937: Neither blocks nor log segments found. Creating new log.
I20260807 07:02:31.782841 1700 log.cc:824] T 00000000000000000000000000000000 P 73652b2d3c38462db13ad7e166a07937: Log is configured to *not* fsync() on all Append() calls
I20260807 07:02:31.785933 1700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73652b2d3c38462db13ad7e166a07937: No bootstrap required, opened a new log
Aug 07 07:02:31 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:02:31.796074 1700 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73652b2d3c38462db13ad7e166a07937 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73652b2d3c38462db13ad7e166a07937" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 38779 } }
I20260807 07:02:31.796545 1700 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73652b2d3c38462db13ad7e166a07937 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:02:31.796685 1700 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73652b2d3c38462db13ad7e166a07937 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73652b2d3c38462db13ad7e166a07937, State: Initialized, Role: FOLLOWER
I20260807 07:02:31.796800 1705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:31.773397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34120 (local address 127.21.83.62:38779)
0807 07:02:31.774283 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:31.774302 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:31.774356 (+ 54us) server_negotiation.cc:407] Connection header received
0807 07:02:31.776817 (+ 2461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:31.776846 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:31.777373 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:31.777786 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:31.779200 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:31.781289 (+ 2089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:31.782571 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:31.783633 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:31.783770 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:31.788210 (+ 4440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:31.788224 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:31.788243 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:31.788292 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:31.792310 (+ 4018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:31.793019 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:31.793028 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:31.793037 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:31.793250 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:31.793599 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:31.793604 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:31.793608 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:31.794381 (+ 773us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:31.794677 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:31.795027 (+ 350us) server_negotiation.cc:299] Negotiation successful
0807 07:02:31.795470 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":331,"threads_started":1}
I20260807 07:02:31.797364 1700 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73652b2d3c38462db13ad7e166a07937 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73652b2d3c38462db13ad7e166a07937" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 38779 } }
I20260807 07:02:31.797626 1700 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73652b2d3c38462db13ad7e166a07937 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:02:31.797753 1700 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73652b2d3c38462db13ad7e166a07937 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:02:31.797921 1700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73652b2d3c38462db13ad7e166a07937 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:02:31.800354 1700 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73652b2d3c38462db13ad7e166a07937 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73652b2d3c38462db13ad7e166a07937" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 38779 } }
I20260807 07:02:31.800823 1700 leader_election.cc:302] T 00000000000000000000000000000000 P 73652b2d3c38462db13ad7e166a07937 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73652b2d3c38462db13ad7e166a07937; no voters:
I20260807 07:02:31.801396 1700 leader_election.cc:288] T 00000000000000000000000000000000 P 73652b2d3c38462db13ad7e166a07937 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 07:02:31.801519 1707 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73652b2d3c38462db13ad7e166a07937 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:02:31.802306 1707 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73652b2d3c38462db13ad7e166a07937 [term 1 LEADER]: Becoming Leader. State: Replica: 73652b2d3c38462db13ad7e166a07937, State: Running, Role: LEADER
I20260807 07:02:31.802814 1707 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73652b2d3c38462db13ad7e166a07937 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73652b2d3c38462db13ad7e166a07937" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 38779 } }
I20260807 07:02:31.803355 1700 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73652b2d3c38462db13ad7e166a07937 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:02:31.805369 1708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73652b2d3c38462db13ad7e166a07937 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73652b2d3c38462db13ad7e166a07937" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73652b2d3c38462db13ad7e166a07937" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 38779 } } }
I20260807 07:02:31.806206 1708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73652b2d3c38462db13ad7e166a07937 [sys.catalog]: This master's current role is: LEADER
I20260807 07:02:31.808935 1711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73652b2d3c38462db13ad7e166a07937 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73652b2d3c38462db13ad7e166a07937. Latest consensus state: current_term: 1 leader_uuid: "73652b2d3c38462db13ad7e166a07937" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73652b2d3c38462db13ad7e166a07937" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 38779 } } }
I20260807 07:02:31.809276 1711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73652b2d3c38462db13ad7e166a07937 [sys.catalog]: This master's current role is: LEADER
I20260807 07:02:31.809932 1720 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:02:31.814109 1720 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:02:31.821818 1720 catalog_manager.cc:1347] Generated new cluster ID: a92b8a3d73814b5889883f57ac65bf1b
I20260807 07:02:31.821934 1720 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:02:31.861332 1720 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:02:31.863250 1720 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:02:31.870554 1720 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73652b2d3c38462db13ad7e166a07937: Generated new TSK 0
I20260807 07:02:31.871340 1720 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.83.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab.
Entry for principal kudu/127.21.83.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab.
I20260807 07:02:31.942397 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.1:0
--local_ip_for_outbound_sockets=127.21.83.1
--webserver_interface=127.21.83.1
--webserver_port=0
--tserver_master_addrs=127.21.83.62:38779
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab
--principal=kudu/127.21.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:34761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:02:32.277987 1737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:02:32.278301 1737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:02:32.278400 1737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:02:32.286309 1737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:02:32.286437 1737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:02:32.286556 1737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.1
I20260807 07:02:32.297134 1737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:34761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab
--principal=kudu/127.21.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.83.1
--webserver_port=0
--tserver_master_addrs=127.21.83.62:38779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.1
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:02:32.298377 1737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:02:32.299619 1737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:02:32.312013 1743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:32.312104 1742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:32.312443 1745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:02:32.312950 1737 server_base.cc:1034] running on GCE node
I20260807 07:02:32.313561 1737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:02:32.314433 1737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:02:32.315624 1737 hybrid_clock.cc:648] HybridClock initialized: now 1786086152315573 us; error 47 us; skew 500 ppm
I20260807 07:02:32.315932 1737 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:02:32 dist-test-slave-ncp9 krb5kdc[1581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086152, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:02:32.318876 1737 init.cc:377] Logged in from keytab as kudu/127.21.83.1@KRBTEST.COM (short username kudu)
I20260807 07:02:32.320206 1737 webserver.cc:460] Webserver started at http://127.21.83.1:32859/ using document root <none> and password file <none>
I20260807 07:02:32.320859 1737 fs_manager.cc:359] Metadata directory not provided
I20260807 07:02:32.321017 1737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:02:32.321305 1737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:02:32.324083 1737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-0/data/instance:
uuid: "815fde3604b14c0bb6d18d377f2da480"
format_stamp: "Formatted at 2026-08-07 07:02:32 on dist-test-slave-ncp9"
server_key: "b7819df959f1a1a0a155ac7bab33ca19"
server_key_iv: "c5469bd9c1fc44ec03dd3b69aa4400cf"
server_key_version: "encryptionkey@0"
I20260807 07:02:32.324858 1737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-0/wal/instance:
uuid: "815fde3604b14c0bb6d18d377f2da480"
format_stamp: "Formatted at 2026-08-07 07:02:32 on dist-test-slave-ncp9"
server_key: "b7819df959f1a1a0a155ac7bab33ca19"
server_key_iv: "c5469bd9c1fc44ec03dd3b69aa4400cf"
server_key_version: "encryptionkey@0"
I20260807 07:02:32.329906 1737 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 07:02:32.333287 1755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:32.334620 1737 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 07:02:32.334779 1737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-0/wal
uuid: "815fde3604b14c0bb6d18d377f2da480"
format_stamp: "Formatted at 2026-08-07 07:02:32 on dist-test-slave-ncp9"
server_key: "b7819df959f1a1a0a155ac7bab33ca19"
server_key_iv: "c5469bd9c1fc44ec03dd3b69aa4400cf"
server_key_version: "encryptionkey@0"
I20260807 07:02:32.334964 1737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:02:32.361769 1737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:02:32.366119 1737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:02:32.366415 1737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:02:32.367409 1737 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:02:32.368882 1737 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:02:32.368986 1737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260807 07:02:32.369125 1737 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:02:32.369181 1737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:32.400019 1737 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.1:45487
I20260807 07:02:32.400069 1905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.1:45487 every 8 connection(s)
I20260807 07:02:32.401245 1737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
I20260807 07:02:32.405233 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 1737
I20260807 07:02:32.405457 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-0/wal/instance
I20260807 07:02:32.406256 21836 external_mini_cluster.cc:1428] Setting key 9dabb7d373db8b8a8b7f86518119e033
Aug 07 07:02:32 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086152, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.1@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:02:32.422529 1910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:32.403643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:35245 (local address 127.21.83.62:38779)
0807 07:02:32.404092 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:32.404097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:32.405124 (+ 1027us) server_negotiation.cc:407] Connection header received
0807 07:02:32.407537 (+ 2413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:32.407542 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:32.407610 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:32.407715 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:32.409651 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:32.410369 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:32.412409 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:32.412803 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:32.412932 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:32.416838 (+ 3906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:32.416843 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:32.416846 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:32.416879 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:32.419380 (+ 2501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:32.420176 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:32.420182 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:32.420185 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:32.420328 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:32.420745 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:32.420749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:32.420752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:32.421106 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:32.421259 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:32.421940 (+ 681us) server_negotiation.cc:299] Negotiation successful
0807 07:02:32.422164 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":218,"threads_started":1}
I20260807 07:02:32.423285 1911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:32.404125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.1:35245)
0807 07:02:32.404930 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:32.404989 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:32.407278 (+ 2289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:32.407957 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:32.407967 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:32.408641 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:32.409452 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:32.409477 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:32.410540 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:32.410546 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:32.412284 (+ 1738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:32.412305 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:32.413054 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:32.413061 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:32.413321 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:32.414004 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:32.414041 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:32.416627 (+ 2586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:32.419510 (+ 2883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:32.419518 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:32.419532 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:32.420056 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:32.420426 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:32.420430 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:32.420434 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:32.420638 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:32.421215 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:32.421220 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:32.421519 (+ 299us) client_negotiation.cc:769] Sending connection context
0807 07:02:32.421782 (+ 263us) client_negotiation.cc:240] Negotiation successful
0807 07:02:32.422181 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":289,"threads_started":1}
I20260807 07:02:32.424817 1906 heartbeater.cc:347] Connected to a master server at 127.21.83.62:38779
I20260807 07:02:32.425220 1906 heartbeater.cc:464] Registering TS with master...
I20260807 07:02:32.426044 1906 heartbeater.cc:511] Master 127.21.83.62:38779 requested a full tablet report, sending...
I20260807 07:02:32.428174 1648 ts_manager.cc:194] Registered new tserver with Master: 815fde3604b14c0bb6d18d377f2da480 (127.21.83.1:45487)
I20260807 07:02:32.429625 1648 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.1@KRBTEST.COM'} at 127.21.83.1:35245
WARNING: no policy specified for kudu/127.21.83.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab.
Entry for principal kudu/127.21.83.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab.
I20260807 07:02:32.464797 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.2:0
--local_ip_for_outbound_sockets=127.21.83.2
--webserver_interface=127.21.83.2
--webserver_port=0
--tserver_master_addrs=127.21.83.62:38779
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab
--principal=kudu/127.21.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:34761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:02:32.796799 1914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:02:32.797081 1914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:02:32.797215 1914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:02:32.804313 1914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:02:32.804410 1914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:02:32.804559 1914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.2
I20260807 07:02:32.813859 1914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:34761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab
--principal=kudu/127.21.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.83.2
--webserver_port=0
--tserver_master_addrs=127.21.83.62:38779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.2
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:02:32.815009 1914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:02:32.816298 1914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:02:32.828677 1924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:32.828702 1926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:32.828595 1928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:02:32.829609 1914 server_base.cc:1034] running on GCE node
I20260807 07:02:32.830191 1914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:02:32.831094 1914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:02:32.832288 1914 hybrid_clock.cc:648] HybridClock initialized: now 1786086152832238 us; error 38 us; skew 500 ppm
I20260807 07:02:32.832566 1914 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:02:32 dist-test-slave-ncp9 krb5kdc[1581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086152, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:02:32.835490 1914 init.cc:377] Logged in from keytab as kudu/127.21.83.2@KRBTEST.COM (short username kudu)
I20260807 07:02:32.836872 1914 webserver.cc:460] Webserver started at http://127.21.83.2:33449/ using document root <none> and password file <none>
I20260807 07:02:32.837550 1914 fs_manager.cc:359] Metadata directory not provided
I20260807 07:02:32.837666 1914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:02:32.837949 1914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:02:32.840828 1914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-1/data/instance:
uuid: "654bbec4196442aa9612d29024ca8047"
format_stamp: "Formatted at 2026-08-07 07:02:32 on dist-test-slave-ncp9"
server_key: "57087c4b46e98430059991a6b2293829"
server_key_iv: "0d05de5f3b972865b4cf02bb6e8b2eb5"
server_key_version: "encryptionkey@0"
I20260807 07:02:32.841607 1914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-1/wal/instance:
uuid: "654bbec4196442aa9612d29024ca8047"
format_stamp: "Formatted at 2026-08-07 07:02:32 on dist-test-slave-ncp9"
server_key: "57087c4b46e98430059991a6b2293829"
server_key_iv: "0d05de5f3b972865b4cf02bb6e8b2eb5"
server_key_version: "encryptionkey@0"
I20260807 07:02:32.846531 1914 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 07:02:32.849714 1936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:32.850904 1914 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 07:02:32.851051 1914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-1/wal
uuid: "654bbec4196442aa9612d29024ca8047"
format_stamp: "Formatted at 2026-08-07 07:02:32 on dist-test-slave-ncp9"
server_key: "57087c4b46e98430059991a6b2293829"
server_key_iv: "0d05de5f3b972865b4cf02bb6e8b2eb5"
server_key_version: "encryptionkey@0"
I20260807 07:02:32.851240 1914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:02:32.875662 1914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:02:32.879251 1914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:02:32.879510 1914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:02:32.880362 1914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:02:32.881604 1914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:02:32.881681 1914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:32.881798 1914 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:02:32.881857 1914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:32.911474 1914 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.2:36391
I20260807 07:02:32.911521 2102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.2:36391 every 8 connection(s)
I20260807 07:02:32.912778 1914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
I20260807 07:02:32.917614 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 1914
I20260807 07:02:32.917845 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-1/wal/instance
I20260807 07:02:32.918372 21836 external_mini_cluster.cc:1428] Setting key 7d2256616cc3ae1a2fb3bb8c98031203
Aug 07 07:02:32 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086152, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.2@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:02:32.935240 1910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:32.915692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:41531 (local address 127.21.83.62:38779)
0807 07:02:32.915875 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:32.915880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:32.917255 (+ 1375us) server_negotiation.cc:407] Connection header received
0807 07:02:32.919747 (+ 2492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:32.919754 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:32.919835 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:32.919942 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:32.922068 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:32.922794 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:32.924980 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:32.925424 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:32.925522 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:32.929719 (+ 4197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:32.929724 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:32.929727 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:32.929763 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:32.932170 (+ 2407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:32.932987 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:32.932990 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:32.932993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:32.933159 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:32.933554 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:32.933558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:32.933561 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:32.933894 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:32.934064 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:32.934668 (+ 604us) server_negotiation.cc:299] Negotiation successful
0807 07:02:32.934889 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":57}
I20260807 07:02:32.935968 2106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:32.916161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.2:41531)
0807 07:02:32.917054 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:32.917094 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:32.919384 (+ 2290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:32.920162 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:32.920172 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:32.920992 (+ 820us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:32.921871 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:32.921900 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:32.922973 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:32.922979 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:32.924804 (+ 1825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:32.924822 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:32.925666 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:32.925674 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:32.925945 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:32.926724 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:32.926753 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:32.929458 (+ 2705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:32.932308 (+ 2850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:32.932316 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:32.932332 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:32.932872 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:32.933270 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:32.933273 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:32.933275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:32.933452 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:32.934005 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:32.934011 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:32.934304 (+ 293us) client_negotiation.cc:769] Sending connection context
0807 07:02:32.934523 (+ 219us) client_negotiation.cc:240] Negotiation successful
0807 07:02:32.934910 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":370,"threads_started":1}
I20260807 07:02:32.937425 2103 heartbeater.cc:347] Connected to a master server at 127.21.83.62:38779
I20260807 07:02:32.937755 2103 heartbeater.cc:464] Registering TS with master...
I20260807 07:02:32.938649 2103 heartbeater.cc:511] Master 127.21.83.62:38779 requested a full tablet report, sending...
I20260807 07:02:32.940150 1648 ts_manager.cc:194] Registered new tserver with Master: 654bbec4196442aa9612d29024ca8047 (127.21.83.2:36391)
I20260807 07:02:32.940953 1648 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.2@KRBTEST.COM'} at 127.21.83.2:41531
WARNING: no policy specified for kudu/127.21.83.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab.
Entry for principal kudu/127.21.83.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab.
I20260807 07:02:32.978542 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.3:0
--local_ip_for_outbound_sockets=127.21.83.3
--webserver_interface=127.21.83.3
--webserver_port=0
--tserver_master_addrs=127.21.83.62:38779
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab
--principal=kudu/127.21.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:34761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:02:33.310981 2109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:02:33.311305 2109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:02:33.311434 2109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:02:33.319190 2109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:02:33.319334 2109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:02:33.319489 2109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.3
I20260807 07:02:33.329869 2109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:34761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab
--principal=kudu/127.21.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.83.3
--webserver_port=0
--tserver_master_addrs=127.21.83.62:38779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.3
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:02:33.331235 2109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:02:33.332536 2109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:02:33.345641 2118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:33.345956 2121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:33.346071 2119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:02:33.346820 2109 server_base.cc:1034] running on GCE node
I20260807 07:02:33.347415 2109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:02:33.348400 2109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:02:33.349617 2109 hybrid_clock.cc:648] HybridClock initialized: now 1786086153349564 us; error 52 us; skew 500 ppm
I20260807 07:02:33.349911 2109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:02:33 dist-test-slave-ncp9 krb5kdc[1581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086153, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:02:33.353084 2109 init.cc:377] Logged in from keytab as kudu/127.21.83.3@KRBTEST.COM (short username kudu)
I20260807 07:02:33.354664 2109 webserver.cc:460] Webserver started at http://127.21.83.3:33391/ using document root <none> and password file <none>
I20260807 07:02:33.355427 2109 fs_manager.cc:359] Metadata directory not provided
I20260807 07:02:33.355556 2109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:02:33.355861 2109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:02:33.358999 2109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-2/data/instance:
uuid: "92b4289f74d641f9bdb897ccbc0fb253"
format_stamp: "Formatted at 2026-08-07 07:02:33 on dist-test-slave-ncp9"
server_key: "43b531c5a8f95e4d6229c1753db9487a"
server_key_iv: "f3f5f24d0da6dab76de08400c62e801f"
server_key_version: "encryptionkey@0"
I20260807 07:02:33.359894 2109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-2/wal/instance:
uuid: "92b4289f74d641f9bdb897ccbc0fb253"
format_stamp: "Formatted at 2026-08-07 07:02:33 on dist-test-slave-ncp9"
server_key: "43b531c5a8f95e4d6229c1753db9487a"
server_key_iv: "f3f5f24d0da6dab76de08400c62e801f"
server_key_version: "encryptionkey@0"
I20260807 07:02:33.365780 2109 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 07:02:33.369576 2129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:33.370976 2109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260807 07:02:33.371153 2109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-2/wal
uuid: "92b4289f74d641f9bdb897ccbc0fb253"
format_stamp: "Formatted at 2026-08-07 07:02:33 on dist-test-slave-ncp9"
server_key: "43b531c5a8f95e4d6229c1753db9487a"
server_key_iv: "f3f5f24d0da6dab76de08400c62e801f"
server_key_version: "encryptionkey@0"
I20260807 07:02:33.371371 2109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:02:33.388053 2109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:02:33.392421 2109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:02:33.392721 2109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:02:33.393833 2109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:02:33.395291 2109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:02:33.395375 2109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:33.395507 2109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:02:33.395561 2109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260807 07:02:33.405623 1901 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.21.83.1:45487
I20260807 07:02:33.430040 2109 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.3:34689
I20260807 07:02:33.430064 2293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.3:34689 every 8 connection(s)
I20260807 07:02:33.431365 2109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
I20260807 07:02:33.432873 1906 heartbeater.cc:503] Master 127.21.83.62:38779 was elected leader, sending a full tablet report...
I20260807 07:02:33.437026 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 2109
I20260807 07:02:33.437203 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-2/wal/instance
I20260807 07:02:33.437625 21836 external_mini_cluster.cc:1428] Setting key 699f1bef82d374674803eb5f17936250
Aug 07 07:02:33 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086153, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.3@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:02:33.454367 1910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:33.433886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:35661 (local address 127.21.83.62:38779)
0807 07:02:33.434067 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:33.434072 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:33.435719 (+ 1647us) server_negotiation.cc:407] Connection header received
0807 07:02:33.437948 (+ 2229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:33.437954 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:33.438085 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:33.438182 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:33.440206 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:33.441147 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:33.442726 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:33.443347 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:33.443480 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:33.447986 (+ 4506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:33.447991 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:33.447995 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:33.448031 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:33.451261 (+ 3230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:33.452119 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:33.452125 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:33.452129 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:33.452307 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:33.452758 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:33.452764 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:33.452767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:33.453139 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:33.453274 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:33.453867 (+ 593us) server_negotiation.cc:299] Negotiation successful
0807 07:02:33.454044 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 07:02:33.455209 2297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:33.434591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.3:35661)
0807 07:02:33.435546 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:33.435599 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:33.437662 (+ 2063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:33.438470 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:33.438479 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:33.439227 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:33.440002 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:33.440032 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:33.441300 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:33.441304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:33.442569 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:33.442588 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:33.443581 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:33.443589 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:33.443889 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:33.444753 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:33.444789 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:33.447720 (+ 2931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:33.451395 (+ 3675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:33.451405 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:33.451418 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:33.451977 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:33.452422 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:33.452426 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:33.452428 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:33.452646 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:33.453248 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:33.453254 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:33.453548 (+ 294us) client_negotiation.cc:769] Sending connection context
0807 07:02:33.453763 (+ 215us) client_negotiation.cc:240] Negotiation successful
0807 07:02:33.454118 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":380,"threads_started":1}
I20260807 07:02:33.456442 2294 heartbeater.cc:347] Connected to a master server at 127.21.83.62:38779
I20260807 07:02:33.456718 2294 heartbeater.cc:464] Registering TS with master...
I20260807 07:02:33.457412 2294 heartbeater.cc:511] Master 127.21.83.62:38779 requested a full tablet report, sending...
I20260807 07:02:33.458703 1648 ts_manager.cc:194] Registered new tserver with Master: 92b4289f74d641f9bdb897ccbc0fb253 (127.21.83.3:34689)
I20260807 07:02:33.459473 1648 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.3@KRBTEST.COM'} at 127.21.83.3:35661
I20260807 07:02:33.465428 21836 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.21.83.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.83.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.83.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.1.keytab.
Entry for principal oryx/127.21.83.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.1.keytab.
WARNING: no policy specified for oryx/127.21.83.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.83.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.83.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.2.keytab.
Entry for principal oryx/127.21.83.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.2.keytab.
WARNING: no policy specified for oryx/127.21.83.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.83.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.83.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.3.keytab.
Entry for principal oryx/127.21.83.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.3.keytab.
I20260807 07:02:33.631215 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 1737
Aug 07 07:02:33 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.1@KRBTEST.COM
I20260807 07:02:33.647362 2307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:33.632461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42512 (local address 127.21.83.1:45487)
0807 07:02:33.633188 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:33.633195 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:33.633222 (+ 27us) server_negotiation.cc:407] Connection header received
0807 07:02:33.633282 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:33.633289 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:33.633491 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:33.633622 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:33.634999 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:33.636012 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:33.637450 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:33.638067 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:33.638202 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:33.642090 (+ 3888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:33.642102 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:33.642124 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:33.642164 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:33.644657 (+ 2493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:33.645350 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:33.645357 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:33.645359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:33.645491 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:33.645848 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:33.645851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:33.645853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:33.646190 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:33.646478 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:33.646766 (+ 288us) server_negotiation.cc:299] Negotiation successful
0807 07:02:33.647021 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":358,"threads_started":1}
I20260807 07:02:33.648538 1900 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:02:33.733615 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 1737
I20260807 07:02:33.756788 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 1914
Aug 07 07:02:33 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.2@KRBTEST.COM
I20260807 07:02:33.773705 2309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:33.757757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56930 (local address 127.21.83.2:36391)
0807 07:02:33.758323 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:33.758330 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:33.758356 (+ 26us) server_negotiation.cc:407] Connection header received
0807 07:02:33.758416 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:33.758422 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:33.758622 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:33.758749 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:33.759970 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:33.761596 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:33.762896 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:33.763718 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:33.763845 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:33.768126 (+ 4281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:33.768140 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:33.768155 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:33.768189 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:33.770995 (+ 2806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:33.771700 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:33.771707 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:33.771709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:33.771851 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:33.772184 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:33.772187 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:33.772188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:33.772561 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:33.772815 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:33.773088 (+ 273us) server_negotiation.cc:299] Negotiation successful
0807 07:02:33.773325 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":299,"threads_started":1}
I20260807 07:02:33.775171 2096 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:02:33.857542 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 1914
I20260807 07:02:33.876556 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 2109
Aug 07 07:02:33 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.3@KRBTEST.COM
I20260807 07:02:33.891640 2313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:33.877590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56234 (local address 127.21.83.3:34689)
0807 07:02:33.878104 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:33.878111 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:33.878141 (+ 30us) server_negotiation.cc:407] Connection header received
0807 07:02:33.878200 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:33.878207 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:33.878423 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:33.878550 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:33.879805 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:33.880791 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:33.882070 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:33.882602 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:33.882754 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:33.886708 (+ 3954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:33.886719 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:33.886731 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:33.886762 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:33.889428 (+ 2666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:33.889980 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:33.889985 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:33.889988 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:33.890099 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:33.890365 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:33.890368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:33.890370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:33.890699 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:33.890921 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:33.891140 (+ 219us) server_negotiation.cc:299] Negotiation successful
0807 07:02:33.891336 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":228,"threads_started":1}
I20260807 07:02:33.892982 2289 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:02:33.977622 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 2109
I20260807 07:02:34.002667 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 1600
I20260807 07:02:34.004098 1692 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:02:34.105268 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 1600
I20260807 07:02:34.129513 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:38779
--webserver_interface=127.21.83.62
--webserver_port=45487
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:34761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:38779
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:02:34.465714 2319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:02:34.465972 2319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:02:34.466099 2319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:02:34.473031 2319 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:02:34.473131 2319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:02:34.473206 2319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:02:34.473275 2319 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:02:34.473342 2319 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:02:34.485365 2319 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:34761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:38779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:38779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.83.62
--webserver_port=45487
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:02:34.487017 2319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:02:34.488364 2319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:02:34.499511 2328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:34.499478 2329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:34.499488 2331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:02:34.500375 2319 server_base.cc:1034] running on GCE node
I20260807 07:02:34.501112 2319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:02:34.502388 2319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:02:34.503578 2319 hybrid_clock.cc:648] HybridClock initialized: now 1786086154503522 us; error 49 us; skew 500 ppm
I20260807 07:02:34.503858 2319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:02:34 dist-test-slave-ncp9 krb5kdc[1581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086154, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:02:34.506834 2319 init.cc:377] Logged in from keytab as kudu/127.21.83.62@KRBTEST.COM (short username kudu)
I20260807 07:02:34.508162 2319 webserver.cc:460] Webserver started at http://127.21.83.62:45487/ using document root <none> and password file <none>
I20260807 07:02:34.508798 2319 fs_manager.cc:359] Metadata directory not provided
I20260807 07:02:34.508962 2319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:02:34.514025 2319 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260807 07:02:34.516615 2345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:34.517814 2319 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 07:02:34.517970 2319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "73652b2d3c38462db13ad7e166a07937"
format_stamp: "Formatted at 2026-08-07 07:02:31 on dist-test-slave-ncp9"
server_key: "13a463efe0e88c2f993fdd1814bddc1b"
server_key_iv: "abb8062dbcea04a3abec6f7003cf1ddc"
server_key_version: "encryptionkey@0"
I20260807 07:02:34.518899 2319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:02:34.539367 2319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:02:34.543536 2319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:02:34.543779 2319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:02:34.561035 2319 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.62:38779
I20260807 07:02:34.561054 2427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.62:38779 every 8 connection(s)
I20260807 07:02:34.562656 2319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:02:34.565893 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 2319
I20260807 07:02:34.566715 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.1:45487
--local_ip_for_outbound_sockets=127.21.83.1
--tserver_master_addrs=127.21.83.62:38779
--webserver_port=32859
--webserver_interface=127.21.83.1
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab
--principal=kudu/127.21.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:34761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.83.1
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 07:02:34.570106 2428 sys_catalog.cc:263] Verifying existing consensus state
I20260807 07:02:34.573730 2428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73652b2d3c38462db13ad7e166a07937: Bootstrap starting.
I20260807 07:02:34.598008 2428 log.cc:824] T 00000000000000000000000000000000 P 73652b2d3c38462db13ad7e166a07937: Log is configured to *not* fsync() on all Append() calls
I20260807 07:02:34.609369 2428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73652b2d3c38462db13ad7e166a07937: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 07:02:34.610009 2428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73652b2d3c38462db13ad7e166a07937: Bootstrap complete.
I20260807 07:02:34.623428 2428 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73652b2d3c38462db13ad7e166a07937 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73652b2d3c38462db13ad7e166a07937" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 38779 } }
I20260807 07:02:34.624596 2428 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73652b2d3c38462db13ad7e166a07937 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73652b2d3c38462db13ad7e166a07937, State: Initialized, Role: FOLLOWER
I20260807 07:02:34.625361 2428 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73652b2d3c38462db13ad7e166a07937 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73652b2d3c38462db13ad7e166a07937" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 38779 } }
I20260807 07:02:34.625643 2428 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73652b2d3c38462db13ad7e166a07937 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:02:34.625770 2428 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73652b2d3c38462db13ad7e166a07937 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:02:34.625964 2428 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73652b2d3c38462db13ad7e166a07937 [term 1 FOLLOWER]: Advancing to term 2
I20260807 07:02:34.631515 2428 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73652b2d3c38462db13ad7e166a07937 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73652b2d3c38462db13ad7e166a07937" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 38779 } }
I20260807 07:02:34.632125 2428 leader_election.cc:302] T 00000000000000000000000000000000 P 73652b2d3c38462db13ad7e166a07937 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73652b2d3c38462db13ad7e166a07937; no voters:
I20260807 07:02:34.632759 2428 leader_election.cc:288] T 00000000000000000000000000000000 P 73652b2d3c38462db13ad7e166a07937 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260807 07:02:34.632930 2433 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73652b2d3c38462db13ad7e166a07937 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 07:02:34.633991 2433 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73652b2d3c38462db13ad7e166a07937 [term 2 LEADER]: Becoming Leader. State: Replica: 73652b2d3c38462db13ad7e166a07937, State: Running, Role: LEADER
I20260807 07:02:34.634574 2433 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73652b2d3c38462db13ad7e166a07937 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73652b2d3c38462db13ad7e166a07937" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 38779 } }
I20260807 07:02:34.635008 2428 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73652b2d3c38462db13ad7e166a07937 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:02:34.637044 2434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73652b2d3c38462db13ad7e166a07937 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "73652b2d3c38462db13ad7e166a07937" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73652b2d3c38462db13ad7e166a07937" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 38779 } } }
I20260807 07:02:34.637421 2434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73652b2d3c38462db13ad7e166a07937 [sys.catalog]: This master's current role is: LEADER
I20260807 07:02:34.637894 2437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73652b2d3c38462db13ad7e166a07937 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73652b2d3c38462db13ad7e166a07937. Latest consensus state: current_term: 2 leader_uuid: "73652b2d3c38462db13ad7e166a07937" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73652b2d3c38462db13ad7e166a07937" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 38779 } } }
I20260807 07:02:34.638223 2437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73652b2d3c38462db13ad7e166a07937 [sys.catalog]: This master's current role is: LEADER
I20260807 07:02:34.647570 2441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:02:34.654129 2441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:02:34.655933 2441 catalog_manager.cc:1259] Loaded cluster ID: a92b8a3d73814b5889883f57ac65bf1b
I20260807 07:02:34.656104 2441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:02:34.660323 2441 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:02:34.662003 2441 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 73652b2d3c38462db13ad7e166a07937: Loaded TSK: 0
I20260807 07:02:34.663708 2441 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 07:02:34.951967 2431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:02:34.952234 2431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:02:34.952349 2431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:02:34.959388 2431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:02:34.959489 2431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:02:34.959630 2431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.1
I20260807 07:02:34.970005 2431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:34761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.1.keytab
--principal=oryx/127.21.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.1:45487
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.83.1
--webserver_port=32859
--tserver_master_addrs=127.21.83.62:38779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.1
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:02:34.971495 2431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:02:34.973105 2431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:02:34.987260 2464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:34.987174 2466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:34.987879 2469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:02:34.988396 2431 server_base.cc:1034] running on GCE node
I20260807 07:02:34.989158 2431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:02:34.990247 2431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:02:34.991482 2431 hybrid_clock.cc:648] HybridClock initialized: now 1786086154991424 us; error 52 us; skew 500 ppm
I20260807 07:02:34.991816 2431 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:02:34 dist-test-slave-ncp9 krb5kdc[1581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086154, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:02:34.999478 2431 init.cc:377] Logged in from keytab as oryx/127.21.83.1@KRBTEST.COM (short username oryx)
I20260807 07:02:35.001368 2431 webserver.cc:460] Webserver started at http://127.21.83.1:32859/ using document root <none> and password file <none>
I20260807 07:02:35.002117 2431 fs_manager.cc:359] Metadata directory not provided
I20260807 07:02:35.002295 2431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:02:35.008117 2431 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260807 07:02:35.010802 2479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:35.012099 2431 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 07:02:35.012255 2431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-0/wal
uuid: "815fde3604b14c0bb6d18d377f2da480"
format_stamp: "Formatted at 2026-08-07 07:02:32 on dist-test-slave-ncp9"
server_key: "b7819df959f1a1a0a155ac7bab33ca19"
server_key_iv: "c5469bd9c1fc44ec03dd3b69aa4400cf"
server_key_version: "encryptionkey@0"
I20260807 07:02:35.013238 2431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:02:35.034850 2431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:02:35.039213 2431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:02:35.039476 2431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:02:35.040580 2431 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:02:35.042128 2431 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:02:35.042253 2431 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:35.042372 2431 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:02:35.042444 2431 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:35.073976 2431 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.1:45487
I20260807 07:02:35.074033 2652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.1:45487 every 8 connection(s)
I20260807 07:02:35.075358 2431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
I20260807 07:02:35.081928 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 2431
I20260807 07:02:35.082679 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.2:36391
--local_ip_for_outbound_sockets=127.21.83.2
--tserver_master_addrs=127.21.83.62:38779
--webserver_port=33449
--webserver_interface=127.21.83.2
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab
--principal=kudu/127.21.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:34761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.83.2
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 07:02:35 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.1@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:35 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.1@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 07:02:35.094399 2658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:35.078156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.1:42447)
0807 07:02:35.078924 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:35.078960 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:35.080584 (+ 1624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:35.081871 (+ 1287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:35.081882 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:35.082617 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:35.083351 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:35.083380 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:35.085325 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:35.085330 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:35.086961 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:35.086975 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:35.088116 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:35.088123 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:35.088364 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:35.089104 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:35.089139 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:35.093242 (+ 4103us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":326,"threads_started":1}
W20260807 07:02:35.094516 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:35.078130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:42447 (local address 127.21.83.62:38779)
0807 07:02:35.078886 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:35.078900 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:35.079077 (+ 177us) server_negotiation.cc:407] Connection header received
0807 07:02:35.081083 (+ 2006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:35.081095 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:35.081388 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:35.081650 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:35.083574 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:35.085156 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:35.087083 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:35.087882 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:35.088025 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:35.093579 (+ 5554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.1:42447: BlockingRecv error: recv got EOF from 127.21.83.1:42447 (error 108)
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":274,"threads_started":1}
W20260807 07:02:35.095289 2653 heartbeater.cc:644] Failed to heartbeat to 127.21.83.62:38779 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.83.62:38779: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Aug 07 07:02:35 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.1@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:35 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.1@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 07:02:35.103040 2658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:35.095827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.1:56093)
0807 07:02:35.095982 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:35.095998 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:35.096085 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:35.096376 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:35.096381 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:35.096696 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:35.096825 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:35.096833 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:35.097912 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:35.097917 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:35.098784 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:35.098794 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:35.099422 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:35.099427 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:35.099584 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:35.100074 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:35.100091 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:35.102804 (+ 2713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260807 07:02:35.103258 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:35.095981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:56093 (local address 127.21.83.62:38779)
0807 07:02:35.096144 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:35.096150 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:35.096166 (+ 16us) server_negotiation.cc:407] Connection header received
0807 07:02:35.096211 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:35.096214 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:35.096263 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:35.096338 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:35.096967 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:35.097780 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:35.098911 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:35.099290 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:35.099394 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:35.102965 (+ 3571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.1:56093: BlockingRecv error: recv got EOF from 127.21.83.1:56093 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 07 07:02:35 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.1@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:35 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.1@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 07:02:35.110930 2658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:35.103668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.1:59499)
0807 07:02:35.103838 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:35.103857 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:35.103938 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:35.104241 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:35.104244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:35.104492 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:35.104630 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:35.104637 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:35.105947 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:35.105950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:35.106815 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:35.106824 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:35.107701 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:35.107704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:35.107813 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:35.108299 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:35.108316 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:35.110731 (+ 2415us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":56}
W20260807 07:02:35.111200 2653 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 07:02:35.111241 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:35.103800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:59499 (local address 127.21.83.62:38779)
0807 07:02:35.103963 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:35.103969 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:35.103987 (+ 18us) server_negotiation.cc:407] Connection header received
0807 07:02:35.104032 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:35.104037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:35.104118 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:35.104218 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:35.104760 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:35.105817 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:35.106945 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:35.107583 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:35.107712 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:35.110969 (+ 3257us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.1:59499: BlockingRecv error: recv got EOF from 127.21.83.1:59499 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260807 07:02:35.413653 2660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:02:35.413938 2660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:02:35.414089 2660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:02:35.421247 2660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:02:35.421355 2660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:02:35.421523 2660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.2
I20260807 07:02:35.431128 2660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:34761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.2.keytab
--principal=oryx/127.21.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.2:36391
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.83.2
--webserver_port=33449
--tserver_master_addrs=127.21.83.62:38779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.2
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:02:35.432374 2660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:02:35.433600 2660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:02:35.446383 2665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:35.446393 2666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:35.446650 2669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:02:35.448300 2660 server_base.cc:1034] running on GCE node
I20260807 07:02:35.449009 2660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:02:35.449889 2660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:02:35.451087 2660 hybrid_clock.cc:648] HybridClock initialized: now 1786086155451039 us; error 40 us; skew 500 ppm
I20260807 07:02:35.451382 2660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:02:35 dist-test-slave-ncp9 krb5kdc[1581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086155, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:02:35.454352 2660 init.cc:377] Logged in from keytab as oryx/127.21.83.2@KRBTEST.COM (short username oryx)
I20260807 07:02:35.455601 2660 webserver.cc:460] Webserver started at http://127.21.83.2:33449/ using document root <none> and password file <none>
I20260807 07:02:35.456211 2660 fs_manager.cc:359] Metadata directory not provided
I20260807 07:02:35.456319 2660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:02:35.461153 2660 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260807 07:02:35.463490 2679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:35.464645 2660 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 07:02:35.464776 2660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-1/wal
uuid: "654bbec4196442aa9612d29024ca8047"
format_stamp: "Formatted at 2026-08-07 07:02:32 on dist-test-slave-ncp9"
server_key: "57087c4b46e98430059991a6b2293829"
server_key_iv: "0d05de5f3b972865b4cf02bb6e8b2eb5"
server_key_version: "encryptionkey@0"
I20260807 07:02:35.465672 2660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:02:35.478107 2660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:02:35.481900 2660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:02:35.482110 2660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:02:35.482967 2660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:02:35.484143 2660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:02:35.484228 2660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:35.484308 2660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:02:35.484360 2660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:35.513945 2660 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.2:36391
I20260807 07:02:35.514012 2849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.2:36391 every 8 connection(s)
I20260807 07:02:35.515242 2660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
I20260807 07:02:35.521446 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 2660
I20260807 07:02:35.522065 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.3:34689
--local_ip_for_outbound_sockets=127.21.83.3
--tserver_master_addrs=127.21.83.62:38779
--webserver_port=33391
--webserver_interface=127.21.83.3
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab
--principal=kudu/127.21.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:34761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.83.3
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 07:02:35 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.2@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:35 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.2@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
W20260807 07:02:35.530217 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:35.517509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:59793 (local address 127.21.83.62:38779)
0807 07:02:35.517708 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:35.517713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:35.518820 (+ 1107us) server_negotiation.cc:407] Connection header received
0807 07:02:35.520567 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:35.520573 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:35.520640 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:35.520725 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:35.522382 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:35.523195 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:35.524746 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:35.525211 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:35.525322 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:35.530026 (+ 4704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.2:59793: BlockingRecv error: recv got EOF from 127.21.83.2:59793 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 07:02:35.530968 2855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:35.517996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.2:59793)
0807 07:02:35.518676 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:35.518711 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:35.520318 (+ 1607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:35.520956 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:35.520967 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:35.521621 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:35.522207 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:35.522225 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:35.523357 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:35.523361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:35.524603 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:35.524617 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:35.525419 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:35.525426 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:35.525657 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:35.526250 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:35.526271 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:35.529875 (+ 3604us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":264,"threads_started":1}
W20260807 07:02:35.531603 2850 heartbeater.cc:644] Failed to heartbeat to 127.21.83.62:38779 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.83.62:38779: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Aug 07 07:02:35 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.2@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:35 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.2@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 07:02:35.539211 2855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:35.532123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.2:38889)
0807 07:02:35.532295 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:35.532314 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:35.532403 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:35.532676 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:35.532680 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:35.533074 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:35.533228 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:35.533238 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:35.534243 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:35.534246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:35.535123 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:35.535134 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:35.535930 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:35.535935 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:35.536059 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:35.536578 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:35.536594 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:35.539027 (+ 2433us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260807 07:02:35.539454 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:35.532199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:38889 (local address 127.21.83.62:38779)
0807 07:02:35.532336 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:35.532340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:35.532354 (+ 14us) server_negotiation.cc:407] Connection header received
0807 07:02:35.532496 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:35.532499 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:35.532554 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:35.532645 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:35.533343 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:35.534118 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:35.535246 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:35.535804 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:35.535924 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:35.539248 (+ 3324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.2:38889: BlockingRecv error: recv got EOF from 127.21.83.2:38889 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 07 07:02:35 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.2@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:35 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.2@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 07:02:35.547096 2855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:35.539812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.2:42345)
0807 07:02:35.540048 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:35.540064 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:35.540145 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:35.540381 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:35.540384 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:35.540626 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:35.540757 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:35.540764 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:35.541955 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:35.541958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:35.542820 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:35.542831 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:35.543576 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:35.543579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:35.543712 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:35.544215 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:35.544229 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:35.546808 (+ 2579us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136}
W20260807 07:02:35.547101 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:35.539939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:42345 (local address 127.21.83.62:38779)
0807 07:02:35.540105 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:35.540110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:35.540127 (+ 17us) server_negotiation.cc:407] Connection header received
0807 07:02:35.540216 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:35.540221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:35.540281 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:35.540364 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:35.540884 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:35.541846 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:35.542948 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:35.543463 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:35.543590 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:35.546955 (+ 3365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.2:42345: BlockingRecv error: recv got EOF from 127.21.83.2:42345 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260807 07:02:35.547480 2850 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 07:02:35.864250 2857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:02:35.864504 2857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:02:35.864629 2857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:02:35.871656 2857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:02:35.871779 2857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:02:35.871937 2857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.3
I20260807 07:02:35.882182 2857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:34761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/krb5kdc/oryx_127.21.83.3.keytab
--principal=oryx/127.21.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.3:34689
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.83.3
--webserver_port=33391
--tserver_master_addrs=127.21.83.62:38779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.3
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:02:35.883618 2857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:02:35.884974 2857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:02:35.897696 2864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:35.897727 2867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:35.897655 2865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:02:35.898407 2857 server_base.cc:1034] running on GCE node
I20260807 07:02:35.899016 2857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:02:35.899897 2857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:02:35.901103 2857 hybrid_clock.cc:648] HybridClock initialized: now 1786086155901044 us; error 39 us; skew 500 ppm
I20260807 07:02:35.901391 2857 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:02:35 dist-test-slave-ncp9 krb5kdc[1581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086155, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:02:35.904330 2857 init.cc:377] Logged in from keytab as oryx/127.21.83.3@KRBTEST.COM (short username oryx)
I20260807 07:02:35.905709 2857 webserver.cc:460] Webserver started at http://127.21.83.3:33391/ using document root <none> and password file <none>
I20260807 07:02:35.906352 2857 fs_manager.cc:359] Metadata directory not provided
I20260807 07:02:35.906476 2857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:02:35.911441 2857 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260807 07:02:35.913973 2879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:35.915122 2857 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260807 07:02:35.915266 2857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-2/wal
uuid: "92b4289f74d641f9bdb897ccbc0fb253"
format_stamp: "Formatted at 2026-08-07 07:02:33 on dist-test-slave-ncp9"
server_key: "43b531c5a8f95e4d6229c1753db9487a"
server_key_iv: "f3f5f24d0da6dab76de08400c62e801f"
server_key_version: "encryptionkey@0"
I20260807 07:02:35.916193 2857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:02:35.937615 2857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:02:35.941859 2857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:02:35.942126 2857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:02:35.943156 2857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:02:35.944473 2857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:02:35.944559 2857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:35.944643 2857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:02:35.944698 2857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:35.976663 2857 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.3:34689
I20260807 07:02:35.976738 3042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.3:34689 every 8 connection(s)
I20260807 07:02:35.978015 2857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
I20260807 07:02:35.983214 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 2857
Aug 07 07:02:35 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.3@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:35 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.3@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
W20260807 07:02:35.993697 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:35.980314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:55041 (local address 127.21.83.62:38779)
0807 07:02:35.980490 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:35.980495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:35.981578 (+ 1083us) server_negotiation.cc:407] Connection header received
0807 07:02:35.983542 (+ 1964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:35.983547 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:35.983622 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:35.983703 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:35.985541 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:35.986241 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:35.987943 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:35.988345 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:35.988433 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:35.993399 (+ 4966us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.3:55041: BlockingRecv error: recv got EOF from 127.21.83.3:55041 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 07:02:35.994014 3046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:35.980731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.3:55041)
0807 07:02:35.981427 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:35.981468 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:35.983267 (+ 1799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:35.983930 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:35.983940 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:35.984625 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:35.985369 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:35.985396 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:35.986392 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:35.986397 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:35.987812 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:35.987826 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:35.988553 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:35.988561 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:35.988832 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:35.989489 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:35.989517 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:35.993139 (+ 3622us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":298,"threads_started":1}
W20260807 07:02:35.994752 3043 heartbeater.cc:644] Failed to heartbeat to 127.21.83.62:38779 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.83.62:38779: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
I20260807 07:02:35.998773 3049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:35.984237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34130 (local address 127.21.83.62:38779)
0807 07:02:35.984929 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:35.984934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:35.984950 (+ 16us) server_negotiation.cc:407] Connection header received
0807 07:02:35.984992 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:35.984995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:35.985045 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:35.985132 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:35.986741 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:35.987457 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:35.988952 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:35.989336 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:35.989440 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:35.992146 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:35.992153 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:35.992166 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:35.992198 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:35.995700 (+ 3502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:35.996311 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:35.996320 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:35.996328 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:35.996488 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:35.996850 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:35.996855 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:35.996859 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:35.997320 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:35.997595 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:35.997998 (+ 403us) server_negotiation.cc:299] Negotiation successful
0807 07:02:35.998349 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":470,"threads_started":1}
Aug 07 07:02:35 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.3@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:35 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.3@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 07:02:36.002748 3046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:35.995298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.3:45919)
0807 07:02:35.995463 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:35.995480 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:35.995574 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:35.995924 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:35.995929 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:35.996228 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:35.996353 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:35.996360 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:35.997342 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:35.997346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:35.998190 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:35.998203 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:35.999177 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:35.999181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:35.999302 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:35.999757 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:35.999773 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:36.002567 (+ 2794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260807 07:02:36.002903 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:35.995550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:45919 (local address 127.21.83.62:38779)
0807 07:02:35.995689 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:35.995693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:35.995707 (+ 14us) server_negotiation.cc:407] Connection header received
0807 07:02:35.995746 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:35.995749 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:35.995796 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:35.995877 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:35.996483 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:35.997218 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:35.998376 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:35.999035 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:35.999153 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:36.002724 (+ 3571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.3:45919: BlockingRecv error: recv got EOF from 127.21.83.3:45919 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 07 07:02:36 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.3@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:36 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.3@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 07:02:36.009718 3046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:36.003321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.3:58901)
0807 07:02:36.003476 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:36.003494 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:36.003580 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:36.003789 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:36.003791 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:36.004044 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:36.004176 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:36.004183 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:36.005104 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:36.005107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:36.005966 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:36.005976 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:36.006519 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:36.006522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:36.006634 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:36.007076 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:36.007091 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:36.009504 (+ 2413us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260807 07:02:36.009951 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:36.003395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:58901 (local address 127.21.83.62:38779)
0807 07:02:36.003594 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:36.003598 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:36.003620 (+ 22us) server_negotiation.cc:407] Connection header received
0807 07:02:36.003654 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:36.003657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:36.003701 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:36.003767 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:36.004270 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:36.005010 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:36.006068 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:36.006427 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:36.006508 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:36.009720 (+ 3212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.3:58901: BlockingRecv error: recv got EOF from 127.21.83.3:58901 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
W20260807 07:02:36.010005 3043 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 07:02:36.077867 2649 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.21.83.1:45487
Aug 07 07:02:36 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.1@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:36 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.1@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 07:02:36.119935 3050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:36.111761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.1:56365)
0807 07:02:36.112250 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:36.112267 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:36.112397 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:36.112725 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:36.112729 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:36.113243 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:36.113450 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:36.113459 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:36.114433 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:36.114437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:36.115415 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:36.115426 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:36.116063 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:36.116068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:36.116188 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:36.116706 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:36.116723 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:36.119632 (+ 2909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":241,"threads_started":1}
W20260807 07:02:36.120107 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:36.111872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:56365 (local address 127.21.83.62:38779)
0807 07:02:36.112090 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:36.112096 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:36.112346 (+ 250us) server_negotiation.cc:407] Connection header received
0807 07:02:36.112509 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:36.112513 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:36.112572 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:36.112648 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:36.113586 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:36.114297 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:36.115521 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:36.115927 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:36.116012 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:36.119889 (+ 3877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.1:56365: BlockingRecv error: recv got EOF from 127.21.83.1:56365 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Aug 07 07:02:36 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.2@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:36 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.2@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 07:02:36.555697 3051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:36.547983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.2:59171)
0807 07:02:36.548488 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:36.548504 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:36.548614 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:36.548865 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:36.548869 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:36.549390 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:36.549585 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:36.549596 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:36.550531 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:36.550534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:36.551466 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:36.551476 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:36.552092 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:36.552095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:36.552220 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:36.552773 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:36.552792 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:36.555389 (+ 2597us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":276,"threads_started":1}
W20260807 07:02:36.555835 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:36.548026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:59171 (local address 127.21.83.62:38779)
0807 07:02:36.548219 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:36.548225 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:36.548572 (+ 347us) server_negotiation.cc:407] Connection header received
0807 07:02:36.548698 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:36.548701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:36.548759 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:36.548836 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:36.549731 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:36.550418 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:36.551580 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:36.551966 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:36.552058 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:36.555590 (+ 3532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.2:59171: BlockingRecv error: recv got EOF from 127.21.83.2:59171 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 07 07:02:37 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.3@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:37 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.3@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 07:02:37.019908 3053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:37.010759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.3:51745)
0807 07:02:37.011321 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:37.011339 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:37.011506 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:37.011863 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:37.011868 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:37.012573 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:37.012794 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:37.012805 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:37.014082 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:37.014087 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:37.015118 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:37.015134 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:37.015868 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:37.015876 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:37.016031 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:37.016618 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:37.016637 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:37.019653 (+ 3016us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":297,"threads_started":1}
W20260807 07:02:37.020073 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:37.010906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:51745 (local address 127.21.83.62:38779)
0807 07:02:37.011123 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:37.011131 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:37.011432 (+ 301us) server_negotiation.cc:407] Connection header received
0807 07:02:37.011642 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:37.011646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:37.011719 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:37.011812 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:37.012964 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:37.013923 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:37.015259 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:37.015667 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:37.015787 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:37.019859 (+ 4072us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.3:51745: BlockingRecv error: recv got EOF from 127.21.83.3:51745 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 07 07:02:37 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.1@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:37 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.1@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 07:02:37.129004 3055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:37.120940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.1:55333)
0807 07:02:37.121465 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:37.121483 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:37.121597 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:37.121889 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:37.121892 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:37.122341 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:37.122495 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:37.122505 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:37.123681 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:37.123685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:37.124587 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:37.124600 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:37.125297 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:37.125301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:37.125410 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:37.125980 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:37.126001 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:37.128725 (+ 2724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":269,"threads_started":1}
W20260807 07:02:37.129151 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:37.121013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:55333 (local address 127.21.83.62:38779)
0807 07:02:37.121217 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:37.121224 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:37.121556 (+ 332us) server_negotiation.cc:407] Connection header received
0807 07:02:37.121713 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:37.121716 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:37.121778 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:37.121853 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:37.122655 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:37.123517 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:37.124729 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:37.125169 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:37.125272 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:37.128935 (+ 3663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.1:55333: BlockingRecv error: recv got EOF from 127.21.83.1:55333 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 07 07:02:37 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.2@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:37 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.2@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 07:02:37.566073 3056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:37.556943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.2:44405)
0807 07:02:37.557496 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:37.557513 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:37.557640 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:37.557929 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:37.557932 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:37.558342 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:37.558501 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:37.558510 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:37.559617 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:37.559621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:37.560602 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:37.560618 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:37.561537 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:37.561543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:37.561721 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:37.562446 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:37.562469 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:37.565813 (+ 3344us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":302,"threads_started":1}
W20260807 07:02:37.566308 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:37.557087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:44405 (local address 127.21.83.62:38779)
0807 07:02:37.557266 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:37.557270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:37.557589 (+ 319us) server_negotiation.cc:407] Connection header received
0807 07:02:37.557738 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:37.557742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:37.557805 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:37.557900 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:37.558711 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:37.559462 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:37.560772 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:37.561348 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:37.561487 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:37.566086 (+ 4599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.2:44405: BlockingRecv error: recv got EOF from 127.21.83.2:44405 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 07 07:02:38 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.3@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:38 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.3@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 07:02:38.028757 3058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:38.020874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.3:33795)
0807 07:02:38.021563 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:38.021582 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:38.021699 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:38.021956 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:38.021959 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:38.022419 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:38.022586 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:38.022594 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:38.023597 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:38.023600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:38.024478 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:38.024489 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:38.025148 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:38.025152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:38.025260 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:38.025836 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:38.025854 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:38.028507 (+ 2653us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":357,"threads_started":1}
W20260807 07:02:38.028931 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:38.021026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:33795 (local address 127.21.83.62:38779)
0807 07:02:38.021209 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:38.021215 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:38.021658 (+ 443us) server_negotiation.cc:407] Connection header received
0807 07:02:38.021792 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:38.021796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:38.021859 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:38.021928 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:38.022715 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:38.023480 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:38.024592 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:38.025035 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:38.025127 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:38.028681 (+ 3554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.3:33795: BlockingRecv error: recv got EOF from 127.21.83.3:33795 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 07 07:02:38 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.1@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:38 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.1@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 07:02:38.137738 3059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:38.129864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.1:57897)
0807 07:02:38.130376 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:38.130391 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:38.130539 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:38.130855 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:38.130861 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:38.131227 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:38.131397 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:38.131406 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:38.132405 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:38.132410 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:38.133325 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:38.133338 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:38.134005 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:38.134011 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:38.134181 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:38.134742 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:38.134760 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:38.137484 (+ 2724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":258,"threads_started":1}
W20260807 07:02:38.137877 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:38.129950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:57897 (local address 127.21.83.62:38779)
0807 07:02:38.130177 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:38.130183 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:38.130489 (+ 306us) server_negotiation.cc:407] Connection header received
0807 07:02:38.130660 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:38.130665 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:38.130728 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:38.130807 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:38.131547 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:38.132270 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:38.133460 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:38.133865 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:38.133983 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:38.137662 (+ 3679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.1:57897: BlockingRecv error: recv got EOF from 127.21.83.1:57897 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 07 07:02:38 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.2@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:38 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.2@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 07:02:38.575018 3060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:38.567124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.2:54613)
0807 07:02:38.567594 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:38.567621 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:38.567735 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:38.568006 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:38.568009 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:38.568375 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:38.568553 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:38.568562 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:38.569804 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:38.569810 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:38.570778 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:38.570793 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:38.571419 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:38.571423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:38.571556 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:38.572168 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:38.572187 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:38.574773 (+ 2586us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":233,"threads_started":1}
W20260807 07:02:38.575163 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:38.567197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:54613 (local address 127.21.83.62:38779)
0807 07:02:38.567401 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:38.567407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:38.567694 (+ 287us) server_negotiation.cc:407] Connection header received
0807 07:02:38.567827 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:38.567831 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:38.567895 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:38.567974 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:38.568713 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:38.569656 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:38.570908 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:38.571309 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:38.571400 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:38.574960 (+ 3560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.2:54613: BlockingRecv error: recv got EOF from 127.21.83.2:54613 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 07 07:02:39 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.3@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:39 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.3@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 07:02:39.037563 3061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:39.029619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.3:60599)
0807 07:02:39.030128 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:39.030144 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:39.030278 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:39.030618 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:39.030623 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:39.030990 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:39.031180 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:39.031191 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:39.032187 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:39.032192 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:39.033115 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:39.033127 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:39.033795 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:39.033801 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:39.033963 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:39.034499 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:39.034516 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:39.037318 (+ 2802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":257,"threads_started":1}
W20260807 07:02:39.037688 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:39.029725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:60599 (local address 127.21.83.62:38779)
0807 07:02:39.029910 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:39.029915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:39.030227 (+ 312us) server_negotiation.cc:407] Connection header received
0807 07:02:39.030388 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:39.030392 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:39.030469 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:39.030549 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:39.031327 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:39.032049 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:39.033249 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:39.033659 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:39.033765 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:39.037496 (+ 3731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.3:60599: BlockingRecv error: recv got EOF from 127.21.83.3:60599 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 07 07:02:39 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.1@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:39 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.1@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 07:02:39.146273 3062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:39.138653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.1:41997)
0807 07:02:39.139102 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:39.139118 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:39.139231 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:39.139489 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:39.139492 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:39.139874 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:39.140026 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:39.140035 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:39.140981 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:39.140985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:39.141928 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:39.141940 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:39.142640 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:39.142643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:39.142816 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:39.143394 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:39.143412 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:39.146035 (+ 2623us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":223,"threads_started":1}
W20260807 07:02:39.146468 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:39.138728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:41997 (local address 127.21.83.62:38779)
0807 07:02:39.138967 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:39.138973 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:39.139188 (+ 215us) server_negotiation.cc:407] Connection header received
0807 07:02:39.139321 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:39.139324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:39.139386 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:39.139475 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:39.140129 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:39.140870 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:39.142071 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:39.142514 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:39.142637 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:39.146236 (+ 3599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.1:41997: BlockingRecv error: recv got EOF from 127.21.83.1:41997 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Aug 07 07:02:39 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.2@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:39 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.2@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 07:02:39.583695 3063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:39.575861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.2:51463)
0807 07:02:39.576405 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:39.576420 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:39.576550 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:39.576880 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:39.576885 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:39.577286 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:39.577440 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:39.577451 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:39.578441 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:39.578447 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:39.579394 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:39.579407 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:39.580110 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:39.580116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:39.580285 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:39.580795 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:39.580812 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:39.583371 (+ 2559us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":315,"threads_started":1}
W20260807 07:02:39.583807 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:39.575896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:51463 (local address 127.21.83.62:38779)
0807 07:02:39.576067 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:39.576074 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:39.576503 (+ 429us) server_negotiation.cc:407] Connection header received
0807 07:02:39.576661 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:39.576666 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:39.576734 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:39.576810 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:39.577595 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:39.578307 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:39.579536 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:39.579966 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:39.580072 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:39.583595 (+ 3523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.2:51463: BlockingRecv error: recv got EOF from 127.21.83.2:51463 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 07 07:02:40 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.3@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:40 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.3@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 07:02:40.050087 3064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:40.038446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.3:42383)
0807 07:02:40.038961 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:40.038977 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:40.039095 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:40.039349 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:40.039352 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:40.039748 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:40.039908 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:40.039917 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:40.040872 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:40.040876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:40.041865 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:40.041876 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:40.042660 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:40.042665 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:40.042804 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:40.046291 (+ 3487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:40.046312 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:40.049749 (+ 3437us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":290,"threads_started":1}
W20260807 07:02:40.050266 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:40.038503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:42383 (local address 127.21.83.62:38779)
0807 07:02:40.038669 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:40.038673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:40.039051 (+ 378us) server_negotiation.cc:407] Connection header received
0807 07:02:40.039187 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:40.039191 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:40.039254 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:40.039332 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:40.040018 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:40.040765 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:40.042017 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:40.042502 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:40.042603 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:40.050001 (+ 7398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.3:42383: BlockingRecv error: recv got EOF from 127.21.83.3:42383 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 07 07:02:40 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.1@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:40 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.1@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 07:02:40.155393 3065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:40.147093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.1:36001)
0807 07:02:40.147597 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:40.147612 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:40.147736 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:40.148076 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:40.148079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:40.148462 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:40.148655 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:40.148665 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:40.149707 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:40.149711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:40.150609 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:40.150621 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:40.151240 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:40.151244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:40.151362 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:40.151935 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:40.151953 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:40.155038 (+ 3085us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":251,"threads_started":1}
W20260807 07:02:40.155503 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:40.147167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:36001 (local address 127.21.83.62:38779)
0807 07:02:40.147405 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:40.147411 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:40.147694 (+ 283us) server_negotiation.cc:407] Connection header received
0807 07:02:40.147852 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:40.147857 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:40.147923 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:40.148002 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:40.148775 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:40.149559 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:40.150714 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:40.151140 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:40.151225 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:40.155286 (+ 4061us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.1:36001: BlockingRecv error: recv got EOF from 127.21.83.1:36001 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 07 07:02:40 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.2@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:40 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.2@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 07:02:40.593585 3066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:40.584807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.2:50523)
0807 07:02:40.585385 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:40.585402 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:40.585553 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:40.585891 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:40.585899 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:40.586348 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:40.586537 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:40.586548 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:40.587589 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:40.587595 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:40.588521 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:40.588533 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:40.589431 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:40.589437 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:40.589618 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:40.590303 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:40.590328 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:40.593311 (+ 2983us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":271,"threads_started":1}
W20260807 07:02:40.593824 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:40.585098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:50523 (local address 127.21.83.62:38779)
0807 07:02:40.585314 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:40.585321 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:40.585488 (+ 167us) server_negotiation.cc:407] Connection header received
0807 07:02:40.585678 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:40.585682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:40.585748 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:40.585832 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:40.586691 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:40.587448 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:40.588666 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:40.589280 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:40.589396 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:40.593565 (+ 4169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.2:50523: BlockingRecv error: recv got EOF from 127.21.83.2:50523 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 07 07:02:41 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.3@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:41 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.3@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 07:02:41.059928 3069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:41.051232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.3:35533)
0807 07:02:41.051800 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:41.051817 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:41.051943 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:41.052253 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:41.052256 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:41.052662 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:41.052833 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:41.052843 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:41.053959 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:41.053963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:41.055213 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:41.055232 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:41.056137 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:41.056142 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:41.056267 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:41.056843 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:41.056862 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:41.059670 (+ 2808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":288,"threads_started":1}
W20260807 07:02:41.060135 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:41.051367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:35533 (local address 127.21.83.62:38779)
0807 07:02:41.051588 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:41.051594 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:41.051896 (+ 302us) server_negotiation.cc:407] Connection header received
0807 07:02:41.052069 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:41.052074 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:41.052142 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:41.052221 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:41.053011 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:41.053769 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:41.055398 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:41.056003 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:41.056101 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:41.059902 (+ 3801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.3:35533: BlockingRecv error: recv got EOF from 127.21.83.3:35533 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 07 07:02:41 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.1@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:41 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.1@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 07:02:41.165234 3071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:41.156534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.1:46201)
0807 07:02:41.157174 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:41.157195 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:41.157335 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:41.157857 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:41.157862 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:41.158290 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:41.158456 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:41.158464 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:41.159502 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:41.159506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:41.160427 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:41.160440 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:41.161211 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:41.161216 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:41.161353 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:41.161933 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:41.161951 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:41.164973 (+ 3022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":355,"threads_started":1}
W20260807 07:02:41.165437 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:41.156589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:46201 (local address 127.21.83.62:38779)
0807 07:02:41.156784 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:41.156791 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:41.157280 (+ 489us) server_negotiation.cc:407] Connection header received
0807 07:02:41.157463 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:41.157468 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:41.157543 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:41.157762 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:41.158604 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:41.159360 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:41.160585 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:41.161075 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:41.161181 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:41.165208 (+ 4027us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.1:46201: BlockingRecv error: recv got EOF from 127.21.83.1:46201 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 07 07:02:41 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.2@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:41 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.2@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 07:02:41.603274 3072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:41.594582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.2:58561)
0807 07:02:41.595207 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:41.595225 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:41.595381 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:41.595729 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:41.595734 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:41.596173 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:41.596343 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:41.596353 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:41.597483 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:41.597488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:41.598433 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:41.598447 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:41.599207 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:41.599214 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:41.599382 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:41.600086 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:41.600105 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:41.603033 (+ 2928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":362,"threads_started":1}
W20260807 07:02:41.603574 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:41.594786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:58561 (local address 127.21.83.62:38779)
0807 07:02:41.595033 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:41.595040 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:41.595327 (+ 287us) server_negotiation.cc:407] Connection header received
0807 07:02:41.595511 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:41.595517 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:41.595585 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:41.595672 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:41.596525 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:41.597334 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:41.598601 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:41.599034 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:41.599147 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:41.603317 (+ 4170us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.2:58561: BlockingRecv error: recv got EOF from 127.21.83.2:58561 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 07 07:02:42 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.3@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:42 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.3@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 07:02:42.069936 3073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:42.061021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.3:54161)
0807 07:02:42.061583 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:42.061601 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:42.061732 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:42.062109 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:42.062113 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:42.062564 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:42.062741 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:42.062750 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:42.063818 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:42.063822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:42.064735 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:42.064749 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:42.065515 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:42.065520 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:42.065648 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:42.066233 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:42.066254 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:42.069634 (+ 3380us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":310,"threads_started":1}
W20260807 07:02:42.070173 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:42.061094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:54161 (local address 127.21.83.62:38779)
0807 07:02:42.061260 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:42.061264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:42.061700 (+ 436us) server_negotiation.cc:407] Connection header received
0807 07:02:42.061860 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:42.061864 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:42.061934 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:42.062022 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:42.062883 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:42.063652 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:42.064858 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:42.065349 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:42.065446 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:42.069909 (+ 4463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.3:54161: BlockingRecv error: recv got EOF from 127.21.83.3:54161 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 07 07:02:42 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.1@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:42 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.1@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 07:02:42.175418 3074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:42.166327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.1:38407)
0807 07:02:42.167187 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:42.167204 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:42.167353 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:42.167709 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:42.167715 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:42.168201 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:42.168453 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:42.168467 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:42.169587 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:42.169590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:42.170580 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:42.170593 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:42.171333 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:42.171339 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:42.171501 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:42.172158 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:42.172185 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:42.175051 (+ 2866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":751,"thread_start_us":514,"threads_started":1}
W20260807 07:02:42.175524 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:42.166515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:38407 (local address 127.21.83.62:38779)
0807 07:02:42.166798 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:42.166805 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:42.167297 (+ 492us) server_negotiation.cc:407] Connection header received
0807 07:02:42.167476 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:42.167480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:42.167543 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:42.167642 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:42.168616 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:42.169442 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:42.170719 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:42.171156 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:42.171279 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:42.175298 (+ 4019us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.1:38407: BlockingRecv error: recv got EOF from 127.21.83.1:38407 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
Aug 07 07:02:42 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.2@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:42 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.2@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 07:02:42.613018 3075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:42.604351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.2:41631)
0807 07:02:42.604999 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:42.605017 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:42.605152 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:42.605493 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:42.605497 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:42.605914 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:42.606105 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:42.606115 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:42.607257 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:42.607262 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:42.608390 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:42.608409 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:42.609097 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:42.609102 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:42.609219 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:42.609814 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:42.609832 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:42.612699 (+ 2867us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":266,"threads_started":1}
W20260807 07:02:42.613150 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:42.604529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:41631 (local address 127.21.83.62:38779)
0807 07:02:42.604753 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:42.604775 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:42.605089 (+ 314us) server_negotiation.cc:407] Connection header received
0807 07:02:42.605273 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:42.605277 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:42.605348 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:42.605443 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:42.606227 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:42.607072 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:42.608529 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:42.608975 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:42.609067 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:42.612915 (+ 3848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.2:41631: BlockingRecv error: recv got EOF from 127.21.83.2:41631 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Aug 07 07:02:43 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.3@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:43 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.3@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 07:02:43.079668 3076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:43.070909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.3:36577)
0807 07:02:43.071504 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:43.071521 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:43.071661 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:43.071972 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:43.071977 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:43.072427 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:43.072605 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:43.072616 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:43.073798 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:43.073802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:43.074797 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:43.074811 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:43.075574 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:43.075580 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:43.075741 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:43.076410 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:43.076429 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:43.079298 (+ 2869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":241,"threads_started":1}
W20260807 07:02:43.079810 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:43.070969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:36577 (local address 127.21.83.62:38779)
0807 07:02:43.071134 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:43.071138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:43.071605 (+ 467us) server_negotiation.cc:407] Connection header received
0807 07:02:43.071774 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:43.071778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:43.071844 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:43.071924 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:43.072749 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:43.073657 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:43.074935 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:43.075417 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:43.075528 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:43.079571 (+ 4043us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.3:36577: BlockingRecv error: recv got EOF from 127.21.83.3:36577 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 07 07:02:43 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.1@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:43 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.1@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 07:02:43.185017 3077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:43.176616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.1:43439)
0807 07:02:43.177177 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:43.177197 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:43.177341 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:43.177743 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:43.177749 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:43.178270 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:43.178468 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:43.178484 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:43.179603 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:43.179608 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:43.180503 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:43.180515 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:43.181294 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:43.181299 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:43.181419 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:43.181973 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:43.181991 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:43.184754 (+ 2763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":274,"threads_started":1}
W20260807 07:02:43.185226 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:43.176929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:43439 (local address 127.21.83.62:38779)
0807 07:02:43.177167 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:43.177173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:43.177281 (+ 108us) server_negotiation.cc:407] Connection header received
0807 07:02:43.177484 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:43.177489 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:43.177574 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:43.177677 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:43.178615 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:43.179473 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:43.180655 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:43.181142 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:43.181242 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:43.184988 (+ 3746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.1:43439: BlockingRecv error: recv got EOF from 127.21.83.1:43439 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 07 07:02:43 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.2@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:43 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.2@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 07:02:43.623363 3078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:43.614119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.2:58125)
0807 07:02:43.614736 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:43.614753 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:43.614925 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:43.615283 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:43.615289 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:43.615751 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:43.615914 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:43.615925 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:43.617242 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:43.617247 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:43.618224 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:43.618239 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:43.619056 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:43.619062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:43.619245 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:43.619862 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:43.619881 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:43.623029 (+ 3148us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":320,"threads_started":1}
W20260807 07:02:43.623525 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:43.614270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:58125 (local address 127.21.83.62:38779)
0807 07:02:43.614534 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:43.614541 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:43.614844 (+ 303us) server_negotiation.cc:407] Connection header received
0807 07:02:43.615056 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:43.615062 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:43.615143 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:43.615229 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:43.616070 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:43.617075 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:43.618372 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:43.618867 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:43.618981 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:43.623290 (+ 4309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.2:58125: BlockingRecv error: recv got EOF from 127.21.83.2:58125 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Aug 07 07:02:44 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.3@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:44 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.3@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 07:02:44.089195 3079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:44.080722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.3:39803)
0807 07:02:44.081349 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:44.081366 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:44.081488 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:44.081775 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:44.081778 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:44.082220 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:44.082380 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:44.082389 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:44.083414 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:44.083419 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:44.084364 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:44.084379 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:44.085181 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:44.085186 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:44.085329 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:44.085950 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:44.085968 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:44.088939 (+ 2971us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":344,"threads_started":1}
W20260807 07:02:44.089506 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:44.080836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:39803 (local address 127.21.83.62:38779)
0807 07:02:44.081086 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:44.081091 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:44.081447 (+ 356us) server_negotiation.cc:407] Connection header received
0807 07:02:44.081596 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:44.081600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:44.081663 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:44.081747 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:44.082495 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:44.083262 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:44.084510 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:44.084994 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:44.085087 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:44.089207 (+ 4120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.3:39803: BlockingRecv error: recv got EOF from 127.21.83.3:39803 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Aug 07 07:02:44 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.1@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:44 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.1@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 07:02:44.194267 3080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:44.185946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.1:45375)
0807 07:02:44.186500 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:44.186517 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:44.186660 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:44.186977 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:44.186980 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:44.187384 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:44.187554 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:44.187563 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:44.188739 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:44.188744 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:44.189996 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:44.190010 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:44.190723 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:44.190727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:44.190837 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:44.191366 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:44.191383 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:44.194026 (+ 2643us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":312,"threads_started":1}
W20260807 07:02:44.194456 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:44.186053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:45375 (local address 127.21.83.62:38779)
0807 07:02:44.186239 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:44.186244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:44.186621 (+ 377us) server_negotiation.cc:407] Connection header received
0807 07:02:44.186770 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:44.186774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:44.186860 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:44.186946 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:44.187687 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:44.188588 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:44.190152 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:44.190600 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:44.190696 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:44.194257 (+ 3561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.1:45375: BlockingRecv error: recv got EOF from 127.21.83.1:45375 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 07 07:02:44 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.2@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:44 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.2@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 07:02:44.632751 3081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:44.624566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.2:33943)
0807 07:02:44.625098 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:44.625115 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:44.625234 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:44.625502 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:44.625505 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:44.625899 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:44.626087 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:44.626097 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:44.627026 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:44.627029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:44.627943 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:44.627954 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:44.628616 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:44.628620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:44.628743 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:44.629397 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:44.629415 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:44.632486 (+ 3071us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":254,"threads_started":1}
W20260807 07:02:44.632881 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:44.624631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:33943 (local address 127.21.83.62:38779)
0807 07:02:44.624844 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:44.624850 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:44.625190 (+ 340us) server_negotiation.cc:407] Connection header received
0807 07:02:44.625325 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:44.625328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:44.625397 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:44.625475 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:44.626196 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:44.626925 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:44.628078 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:44.628506 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:44.628597 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:44.632678 (+ 4081us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.2:33943: BlockingRecv error: recv got EOF from 127.21.83.2:33943 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 07 07:02:45 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.3@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:45 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.3@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 07:02:45.102074 3083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:45.090367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.3:34323)
0807 07:02:45.090959 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:45.090977 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:45.091142 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:45.091494 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:45.091499 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:45.091930 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:45.092116 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:45.092126 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:45.093206 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:45.093212 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:45.094175 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:45.094188 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:45.094882 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:45.094888 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:45.095065 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:45.098910 (+ 3845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:45.098928 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:45.101725 (+ 2797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":328,"threads_started":1}
W20260807 07:02:45.102152 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:45.090588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:34323 (local address 127.21.83.62:38779)
0807 07:02:45.090800 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:45.090806 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:45.091084 (+ 278us) server_negotiation.cc:407] Connection header received
0807 07:02:45.091259 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:45.091263 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:45.091327 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:45.091414 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:45.092264 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:45.093070 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:45.094317 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:45.094733 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:45.094839 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:45.101938 (+ 7099us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.3:34323: BlockingRecv error: recv got EOF from 127.21.83.3:34323 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Aug 07 07:02:45 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.1@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:45 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.1@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 07:02:45.203557 3084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:45.195203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.1:47255)
0807 07:02:45.195788 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:45.195803 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:45.195916 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:45.196215 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:45.196218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:45.196616 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:45.196765 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:45.196774 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:45.197922 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:45.197927 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:45.199067 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:45.199085 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:45.199759 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:45.199763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:45.199880 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:45.200442 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:45.200460 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:45.203243 (+ 2783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":349,"threads_started":1}
W20260807 07:02:45.203694 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:45.195312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:47255 (local address 127.21.83.62:38779)
0807 07:02:45.195477 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:45.195482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:45.195867 (+ 385us) server_negotiation.cc:407] Connection header received
0807 07:02:45.196024 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:45.196028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:45.196103 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:45.196185 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:45.196877 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:45.197784 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:45.199198 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:45.199639 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:45.199727 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:45.203443 (+ 3716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.1:47255: BlockingRecv error: recv got EOF from 127.21.83.1:47255 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 07 07:02:45 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.2@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:45 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.2@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 07:02:45.641943 3087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:45.633717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.2:52137)
0807 07:02:45.634259 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:45.634275 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:45.634405 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:45.634708 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:45.634712 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:45.635140 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:45.635306 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:45.635316 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:45.636364 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:45.636368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:45.637311 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:45.637324 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:45.638049 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:45.638055 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:45.638216 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:45.638771 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:45.638790 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:45.641619 (+ 2829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":307,"threads_started":1}
W20260807 07:02:45.642088 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:45.633821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:52137 (local address 127.21.83.62:38779)
0807 07:02:45.634054 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:45.634061 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:45.634357 (+ 296us) server_negotiation.cc:407] Connection header received
0807 07:02:45.634512 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:45.634518 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:45.634583 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:45.634664 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:45.635446 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:45.636226 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:45.637443 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:45.637906 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:45.638015 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:45.641814 (+ 3799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.2:52137: BlockingRecv error: recv got EOF from 127.21.83.2:52137 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 07 07:02:46 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.3@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:46 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.3@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 07:02:46.111796 3089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:46.103247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.3:51287)
0807 07:02:46.103835 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:46.103853 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:46.103989 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:46.104353 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:46.104356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:46.104769 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:46.104972 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:46.104981 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:46.106046 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:46.106051 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:46.107029 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:46.107043 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:46.107776 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:46.107781 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:46.107913 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:46.108588 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:46.108607 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:46.111550 (+ 2943us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":329,"threads_started":1}
W20260807 07:02:46.112031 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:46.103517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:51287 (local address 127.21.83.62:38779)
0807 07:02:46.103821 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:46.103827 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:46.103932 (+ 105us) server_negotiation.cc:407] Connection header received
0807 07:02:46.104158 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:46.104162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:46.104233 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:46.104323 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:46.105123 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:46.105882 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:46.107193 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:46.107616 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:46.107717 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:46.111795 (+ 4078us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.3:51287: BlockingRecv error: recv got EOF from 127.21.83.3:51287 (error 108)
Metrics: {"server-negotiator.queue_time_us":145}
Aug 07 07:02:46 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.1@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:46 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.1@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 07:02:46.212206 3090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:46.204462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.1:52695)
0807 07:02:46.205037 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:46.205058 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:46.205198 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:46.205575 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:46.205592 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:46.205999 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:46.206210 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:46.206219 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:46.207203 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:46.207207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:46.208114 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:46.208126 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:46.208764 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:46.208768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:46.208875 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:46.209415 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:46.209432 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:46.211969 (+ 2537us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":242,"threads_started":1}
W20260807 07:02:46.212374 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:46.204545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:52695 (local address 127.21.83.62:38779)
0807 07:02:46.204755 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:46.204761 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:46.205134 (+ 373us) server_negotiation.cc:407] Connection header received
0807 07:02:46.205322 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:46.205328 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:46.205429 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:46.205535 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:46.206342 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:46.207075 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:46.208220 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:46.208655 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:46.208742 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:46.212171 (+ 3429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.1:52695: BlockingRecv error: recv got EOF from 127.21.83.1:52695 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 07 07:02:46 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.2@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:46 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.2@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 07:02:46.651281 3091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:46.643066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.2:52123)
0807 07:02:46.643596 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:46.643612 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:46.643729 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:46.644073 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:46.644077 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:46.644543 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:46.644736 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:46.644746 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:46.645832 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:46.645836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:46.646772 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:46.646787 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:46.647398 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:46.647402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:46.647519 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:46.648125 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:46.648143 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:46.651015 (+ 2872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":287,"threads_started":1}
W20260807 07:02:46.651429 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:46.643119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:52123 (local address 127.21.83.62:38779)
0807 07:02:46.643299 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:46.643305 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:46.643687 (+ 382us) server_negotiation.cc:407] Connection header received
0807 07:02:46.643841 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:46.643844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:46.643916 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:46.643995 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:46.644865 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:46.645657 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:46.646896 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:46.647286 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:46.647370 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:46.651217 (+ 3847us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.2:52123: BlockingRecv error: recv got EOF from 127.21.83.2:52123 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 07 07:02:47 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.3@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:47 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.3@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 07:02:47.121150 3092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:47.112715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.3:59579)
0807 07:02:47.113217 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:47.113234 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:47.113353 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:47.113626 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:47.113629 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:47.114050 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:47.114238 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:47.114246 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:47.115245 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:47.115249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:47.116222 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:47.116238 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:47.117115 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:47.117121 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:47.117292 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:47.118033 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:47.118050 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:47.120837 (+ 2787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":283,"threads_started":1}
W20260807 07:02:47.121239 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:47.112832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:59579 (local address 127.21.83.62:38779)
0807 07:02:47.113055 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:47.113060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:47.113330 (+ 270us) server_negotiation.cc:407] Connection header received
0807 07:02:47.113451 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:47.113455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:47.113518 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:47.113588 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:47.114387 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:47.115121 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:47.116349 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:47.116955 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:47.117066 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:47.121063 (+ 3997us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.3:59579: BlockingRecv error: recv got EOF from 127.21.83.3:59579 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 07 07:02:47 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.1@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:47 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.1@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 07:02:47.221397 3093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:47.213106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.1:52799)
0807 07:02:47.213576 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:47.213591 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:47.213733 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:47.214065 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:47.214070 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:47.214495 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:47.214652 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:47.214662 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:47.215663 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:47.215667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:47.216639 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:47.216653 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:47.217472 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:47.217478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:47.217653 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:47.218205 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:47.218222 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:47.221064 (+ 2842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":266,"threads_started":1}
W20260807 07:02:47.221504 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:47.213160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:52799 (local address 127.21.83.62:38779)
0807 07:02:47.213337 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:47.213343 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:47.213674 (+ 331us) server_negotiation.cc:407] Connection header received
0807 07:02:47.213851 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:47.213856 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:47.213938 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:47.214014 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:47.214795 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:47.215517 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:47.216774 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:47.217331 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:47.217440 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:47.221261 (+ 3821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.1:52799: BlockingRecv error: recv got EOF from 127.21.83.1:52799 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 07 07:02:47 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.2@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:47 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.2@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 07:02:47.659922 3094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:47.652179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.2:48357)
0807 07:02:47.652666 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:47.652681 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:47.652810 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:47.653156 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:47.653161 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:47.653587 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:47.653741 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:47.653751 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:47.654732 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:47.654736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:47.655638 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:47.655651 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:47.656326 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:47.656331 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:47.656516 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:47.657115 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:47.657132 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:47.659596 (+ 2464us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":279,"threads_started":1}
W20260807 07:02:47.660070 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:47.652237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:48357 (local address 127.21.83.62:38779)
0807 07:02:47.652425 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:47.652431 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:47.652763 (+ 332us) server_negotiation.cc:407] Connection header received
0807 07:02:47.652940 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:47.652944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:47.653014 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:47.653093 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:47.653880 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:47.654596 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:47.655772 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:47.656195 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:47.656299 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:47.659818 (+ 3519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.2:48357: BlockingRecv error: recv got EOF from 127.21.83.2:48357 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 07 07:02:48 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.3@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:48 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.3@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 07:02:48.130054 3095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:48.122143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.3:42649)
0807 07:02:48.122671 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:48.122687 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:48.122833 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:48.123159 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:48.123163 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:48.123545 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:48.123705 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:48.123714 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:48.124728 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:48.124732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:48.125682 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:48.125695 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:48.126291 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:48.126295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:48.126415 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:48.126942 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:48.126959 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:48.129801 (+ 2842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":299,"threads_started":1}
W20260807 07:02:48.130172 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:48.122220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:42649 (local address 127.21.83.62:38779)
0807 07:02:48.122371 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:48.122375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:48.122755 (+ 380us) server_negotiation.cc:407] Connection header received
0807 07:02:48.122933 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:48.122936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:48.122997 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:48.123079 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:48.123822 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:48.124590 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:48.125783 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:48.126188 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:48.126276 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:48.129972 (+ 3696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.3:42649: BlockingRecv error: recv got EOF from 127.21.83.3:42649 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 07 07:02:48 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.1@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:48 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.1@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 07:02:48.230611 3097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:48.222617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.1:58167)
0807 07:02:48.223149 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:48.223168 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:48.223306 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:48.223591 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:48.223594 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:48.223980 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:48.224139 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:48.224148 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:48.225159 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:48.225163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:48.226080 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:48.226094 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:48.226801 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:48.226806 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:48.226925 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:48.227495 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:48.227514 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:48.230286 (+ 2772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":251,"threads_started":1}
W20260807 07:02:48.230741 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:48.222684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:58167 (local address 127.21.83.62:38779)
0807 07:02:48.222897 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:48.222903 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:48.223248 (+ 345us) server_negotiation.cc:407] Connection header received
0807 07:02:48.223396 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:48.223400 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:48.223486 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:48.223560 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:48.224265 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:48.225034 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:48.226209 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:48.226639 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:48.226728 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:48.230513 (+ 3785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.1:58167: BlockingRecv error: recv got EOF from 127.21.83.1:58167 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 07 07:02:48 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.2@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:48 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.2@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 07:02:48.669600 3099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:48.661042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.2:46129)
0807 07:02:48.661641 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:48.661658 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:48.661832 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:48.662176 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:48.662181 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:48.662635 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:48.662807 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:48.662817 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:48.663872 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:48.663876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:48.664803 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:48.664816 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:48.665622 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:48.665628 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:48.665789 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:48.666380 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:48.666400 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:48.669274 (+ 2874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":289,"threads_started":1}
W20260807 07:02:48.669737 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:48.661317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:46129 (local address 127.21.83.62:38779)
0807 07:02:48.661533 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:48.661538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:48.661762 (+ 224us) server_negotiation.cc:407] Connection header received
0807 07:02:48.661963 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:48.661968 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:48.662032 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:48.662135 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:48.662966 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:48.663722 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:48.664989 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:48.665431 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:48.665542 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:48.669510 (+ 3968us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.2:46129: BlockingRecv error: recv got EOF from 127.21.83.2:46129 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Aug 07 07:02:49 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.3@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:49 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.3@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 07:02:49.139825 3101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:49.131032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.3:40479)
0807 07:02:49.131609 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:49.131627 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:49.131776 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:49.132121 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:49.132127 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:49.132557 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:49.132731 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:49.132741 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:49.133825 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:49.133829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:49.134794 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:49.134808 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:49.135616 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:49.135623 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:49.135788 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:49.136524 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:49.136544 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:49.139463 (+ 2919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":303,"threads_started":1}
W20260807 07:02:49.139962 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:49.131116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:40479 (local address 127.21.83.62:38779)
0807 07:02:49.131321 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:49.131327 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:49.131720 (+ 393us) server_negotiation.cc:407] Connection header received
0807 07:02:49.131898 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:49.131903 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:49.131977 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:49.132067 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:49.132883 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:49.133673 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:49.134978 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:49.135415 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:49.135546 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:49.139695 (+ 4149us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.3:40479: BlockingRecv error: recv got EOF from 127.21.83.3:40479 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 07 07:02:49 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.1@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:49 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.1@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 07:02:49.241343 3102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:49.231850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.1:36361)
0807 07:02:49.232981 (+ 1131us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:49.233003 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:49.233138 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:49.233521 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:49.233524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:49.233927 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:49.234120 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:49.234130 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:49.235283 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:49.235287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:49.236235 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:49.236248 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:49.237122 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:49.237127 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:49.237258 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:49.237945 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:49.237964 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:49.241060 (+ 3096us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":970,"thread_start_us":835,"threads_started":1}
W20260807 07:02:49.241611 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:49.232002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:36361 (local address 127.21.83.62:38779)
0807 07:02:49.232394 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:49.232399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:49.233079 (+ 680us) server_negotiation.cc:407] Connection header received
0807 07:02:49.233256 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:49.233260 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:49.233361 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:49.233446 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:49.234282 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:49.235084 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:49.236422 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:49.236960 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:49.237058 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:49.241334 (+ 4276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.1:36361: BlockingRecv error: recv got EOF from 127.21.83.1:36361 (error 108)
Metrics: {"server-negotiator.queue_time_us":158}
Aug 07 07:02:49 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.2@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:49 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.2@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 07:02:49.677848 3104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:49.670480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.2:37563)
0807 07:02:49.671020 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:49.671035 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:49.671147 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:49.671401 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:49.671404 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:49.671751 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:49.671929 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:49.671938 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:49.672867 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:49.672872 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:49.673797 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:49.673809 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:49.674428 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:49.674432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:49.674541 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:49.675064 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:49.675080 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:49.677600 (+ 2520us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":335,"threads_started":1}
W20260807 07:02:49.677978 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:49.670595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:37563 (local address 127.21.83.62:38779)
0807 07:02:49.670739 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:49.670744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:49.671114 (+ 370us) server_negotiation.cc:407] Connection header received
0807 07:02:49.671234 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:49.671238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:49.671299 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:49.671374 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:49.672044 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:49.672759 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:49.673905 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:49.674323 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:49.674410 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:49.677801 (+ 3391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.2:37563: BlockingRecv error: recv got EOF from 127.21.83.2:37563 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 07 07:02:50 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.3@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:50 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.3@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 07:02:50.153390 3105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:50.140927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.3:34205)
0807 07:02:50.141408 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:50.141424 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:50.141549 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:50.141880 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:50.141885 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:50.142399 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:50.142652 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:50.142668 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:50.143885 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:50.143891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:50.144808 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:50.144821 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:50.145543 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:50.145549 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:50.145717 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:50.150354 (+ 4637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:50.150373 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:50.153130 (+ 2757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":253,"threads_started":1}
W20260807 07:02:50.153614 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:50.141031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:34205 (local address 127.21.83.62:38779)
0807 07:02:50.141220 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:50.141224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:50.141516 (+ 292us) server_negotiation.cc:407] Connection header received
0807 07:02:50.141644 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:50.141648 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:50.141716 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:50.141812 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:50.142793 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:50.143739 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:50.144968 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:50.145395 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:50.145516 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:50.153365 (+ 7849us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.3:34205: BlockingRecv error: recv got EOF from 127.21.83.3:34205 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 07 07:02:50 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.1@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:50 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.1@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 07:02:50.250506 3106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:50.242498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.1:43747)
0807 07:02:50.242973 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:50.242989 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:50.243100 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:50.243437 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:50.243441 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:50.243888 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:50.244105 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:50.244115 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:50.245139 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:50.245143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:50.246070 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:50.246082 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:50.246712 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:50.246716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:50.246825 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:50.247339 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:50.247356 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:50.250254 (+ 2898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":271,"threads_started":1}
W20260807 07:02:50.250610 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:50.242556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:43747 (local address 127.21.83.62:38779)
0807 07:02:50.242750 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:50.242758 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:50.243071 (+ 313us) server_negotiation.cc:407] Connection header received
0807 07:02:50.243200 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:50.243203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:50.243265 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:50.243339 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:50.244231 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:50.245008 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:50.246184 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:50.246604 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:50.246694 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:50.250418 (+ 3724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.1:43747: BlockingRecv error: recv got EOF from 127.21.83.1:43747 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 07 07:02:50 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.2@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 07:02:50 dist-test-slave-ncp9 krb5kdc[1581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.83.2@KRBTEST.COM for oryx/127.21.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 07:02:50.686781 3107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:50.678898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38779 (local address 127.21.83.2:59073)
0807 07:02:50.679383 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:50.679398 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:50.679509 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:50.679773 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:50.679776 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:50.680181 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:50.680379 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:50.680389 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:50.681360 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:50.681363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:50.682297 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:50.682308 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:50.682966 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:50.682969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:50.683102 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:50.683729 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:50.683751 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:50.686421 (+ 2670us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.83.62:38779: Server oryx/127.21.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":270,"threads_started":1}
W20260807 07:02:50.686882 2656 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:50.678945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:59073 (local address 127.21.83.62:38779)
0807 07:02:50.679144 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:50.679151 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:50.679470 (+ 319us) server_negotiation.cc:407] Connection header received
0807 07:02:50.679612 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:50.679616 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:50.679675 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:50.679743 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:50.680480 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:50.681255 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:50.682405 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:50.682865 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:50.682952 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:50.686656 (+ 3704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.83.2:59073: BlockingRecv error: recv got EOF from 127.21.83.2:59073 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 07:02:50.984732 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 2431
I20260807 07:02:50.997815 3109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:50.985887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50026 (local address 127.21.83.1:45487)
0807 07:02:50.986405 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:50.986414 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:50.986442 (+ 28us) server_negotiation.cc:407] Connection header received
0807 07:02:50.986598 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:50.986607 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:50.986864 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:50.987034 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:50.988550 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:50.989597 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:50.991000 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:50.991606 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:50.991731 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:50.994411 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:50.994422 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:50.994438 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:50.994485 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:50.997229 (+ 2744us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.21.83.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0807 07:02:50.997522 (+ 293us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50026: Request ticket server kudu/127.21.83.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":233,"threads_started":1}
W20260807 07:02:50.997965 3109 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50026: Request ticket server kudu/127.21.83.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260807 07:02:50.998345 21836 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 2431: Not authorized: Client connection negotiation failed: client connection to 127.21.83.1:45487: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.21.83.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260807 07:02:50.998560 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 2431
I20260807 07:02:51.022053 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 2660
I20260807 07:02:51.033707 3110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:51.023207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56180 (local address 127.21.83.2:36391)
0807 07:02:51.023833 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:51.023841 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:51.023872 (+ 31us) server_negotiation.cc:407] Connection header received
0807 07:02:51.023935 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:51.023941 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:51.024146 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:51.024301 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:51.025523 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:51.026625 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:51.027772 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:51.028292 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:51.028387 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:51.030720 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:51.030729 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:51.030742 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:51.030777 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:51.033173 (+ 2396us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.21.83.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0807 07:02:51.033424 (+ 251us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56180: Request ticket server kudu/127.21.83.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":314,"threads_started":1}
W20260807 07:02:51.033754 21836 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 2660: Not authorized: Client connection negotiation failed: client connection to 127.21.83.2:36391: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.21.83.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260807 07:02:51.033810 3110 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56180: Request ticket server kudu/127.21.83.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260807 07:02:51.033932 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 2660
I20260807 07:02:51.056022 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 2857
I20260807 07:02:51.067878 3111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:51.057029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54138 (local address 127.21.83.3:34689)
0807 07:02:51.057631 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:51.057639 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:51.057666 (+ 27us) server_negotiation.cc:407] Connection header received
0807 07:02:51.057727 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:51.057734 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:51.057952 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:51.058107 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:51.059354 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:51.060697 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:51.061969 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:51.062539 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:51.062640 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:51.064998 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:51.065006 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:51.065019 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:51.065058 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:51.067400 (+ 2342us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.21.83.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0807 07:02:51.067648 (+ 248us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54138: Request ticket server kudu/127.21.83.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":296,"threads_started":1}
W20260807 07:02:51.067973 3111 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54138: Request ticket server kudu/127.21.83.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260807 07:02:51.068073 21836 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 2857: Not authorized: Client connection negotiation failed: client connection to 127.21.83.3:34689: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.21.83.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260807 07:02:51.068212 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 2857
I20260807 07:02:51.087772 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 2319
I20260807 07:02:51.089047 2423 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:02:51.206744 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 2319
2026-08-07T07:02:51Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21337 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-07T07:02:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:02:51Z Disabled control of system clock
I20260807 07:02:51.296960 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:36419
--webserver_interface=127.21.83.62
--webserver_port=0
--builtin_ntp_servers=127.21.83.20:35341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:36419
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260807 07:02:51.624589 3118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:02:51.624841 3118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:02:51.624948 3118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:02:51.631866 3118 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:02:51.631964 3118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:02:51.632001 3118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:02:51.632073 3118 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:02:51.632117 3118 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:02:51.640791 3118 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:35341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:36419
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:36419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:02:51.641925 3118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:02:51.643594 3118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:02:51.654657 3126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:51.654695 3127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:51.654731 3129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:02:51.655766 3118 server_base.cc:1034] running on GCE node
I20260807 07:02:51.656603 3118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:02:51.658241 3118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:02:51.659479 3118 hybrid_clock.cc:648] HybridClock initialized: now 1786086171659428 us; error 50 us; skew 500 ppm
I20260807 07:02:51.659888 3118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 07:02:51.662531 3118 webserver.cc:460] Webserver started at http://127.21.83.62:41315/ using document root <none> and password file <none>
I20260807 07:02:51.663297 3118 fs_manager.cc:359] Metadata directory not provided
I20260807 07:02:51.663412 3118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:02:51.663763 3118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:02:51.666894 3118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "297ad69a59194383bb21171ce9c5c479"
format_stamp: "Formatted at 2026-08-07 07:02:51 on dist-test-slave-ncp9"
server_key: "12fe7812327a2323436f0bd7f1758a76"
server_key_iv: "f27c2080edb95c816a4a7c0ad487dfa1"
server_key_version: "encryptionkey@0"
I20260807 07:02:51.667701 3118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "297ad69a59194383bb21171ce9c5c479"
format_stamp: "Formatted at 2026-08-07 07:02:51 on dist-test-slave-ncp9"
server_key: "12fe7812327a2323436f0bd7f1758a76"
server_key_iv: "f27c2080edb95c816a4a7c0ad487dfa1"
server_key_version: "encryptionkey@0"
I20260807 07:02:51.672755 3118 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260807 07:02:51.676054 3136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:51.677378 3118 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 07:02:51.677539 3118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "297ad69a59194383bb21171ce9c5c479"
format_stamp: "Formatted at 2026-08-07 07:02:51 on dist-test-slave-ncp9"
server_key: "12fe7812327a2323436f0bd7f1758a76"
server_key_iv: "f27c2080edb95c816a4a7c0ad487dfa1"
server_key_version: "encryptionkey@0"
I20260807 07:02:51.677731 3118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:02:51.697952 3118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:02:51.698895 3118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:02:51.699184 3118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:02:51.717759 3118 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.62:36419
I20260807 07:02:51.717767 3203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.62:36419 every 8 connection(s)
I20260807 07:02:51.719146 3118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:02:51.722864 3205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:02:51.728644 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 3118
I20260807 07:02:51.728860 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:02:51.729702 21836 external_mini_cluster.cc:1428] Setting key 38d4523818500909694521fddb5fa05c
I20260807 07:02:51.734380 3205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 297ad69a59194383bb21171ce9c5c479: Bootstrap starting.
I20260807 07:02:51.737761 3205 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 297ad69a59194383bb21171ce9c5c479: Neither blocks nor log segments found. Creating new log.
I20260807 07:02:51.738991 3205 log.cc:824] T 00000000000000000000000000000000 P 297ad69a59194383bb21171ce9c5c479: Log is configured to *not* fsync() on all Append() calls
I20260807 07:02:51.742085 3205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 297ad69a59194383bb21171ce9c5c479: No bootstrap required, opened a new log
I20260807 07:02:51.743109 3209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:51.731802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38644 (local address 127.21.83.62:36419)
0807 07:02:51.732777 (+ 975us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:51.732788 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:51.732829 (+ 41us) server_negotiation.cc:407] Connection header received
0807 07:02:51.734473 (+ 1644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:51.734496 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:51.734947 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:51.735237 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:51.735932 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:51.737202 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:51.738965 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:51.739778 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:51.739969 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:51.740995 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:51.741004 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:51.741017 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:02:51.741527 (+ 510us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:51.741598 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:02:51.741627 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:51.741709 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:51.741920 (+ 211us) server_negotiation.cc:299] Negotiation successful
0807 07:02:51.742224 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":355,"threads_started":1}
I20260807 07:02:51.749239 3205 raft_consensus.cc:348] T 00000000000000000000000000000000 P 297ad69a59194383bb21171ce9c5c479 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "297ad69a59194383bb21171ce9c5c479" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 36419 } }
I20260807 07:02:51.749559 3205 raft_consensus.cc:374] T 00000000000000000000000000000000 P 297ad69a59194383bb21171ce9c5c479 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:02:51.749647 3205 raft_consensus.cc:729] T 00000000000000000000000000000000 P 297ad69a59194383bb21171ce9c5c479 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297ad69a59194383bb21171ce9c5c479, State: Initialized, Role: FOLLOWER
I20260807 07:02:51.750128 3205 consensus_queue.cc:260] T 00000000000000000000000000000000 P 297ad69a59194383bb21171ce9c5c479 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "297ad69a59194383bb21171ce9c5c479" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 36419 } }
I20260807 07:02:51.750347 3205 raft_consensus.cc:388] T 00000000000000000000000000000000 P 297ad69a59194383bb21171ce9c5c479 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:02:51.750502 3205 raft_consensus.cc:482] T 00000000000000000000000000000000 P 297ad69a59194383bb21171ce9c5c479 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:02:51.750671 3205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 297ad69a59194383bb21171ce9c5c479 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:02:51.752732 3205 raft_consensus.cc:504] T 00000000000000000000000000000000 P 297ad69a59194383bb21171ce9c5c479 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "297ad69a59194383bb21171ce9c5c479" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 36419 } }
I20260807 07:02:51.753206 3205 leader_election.cc:302] T 00000000000000000000000000000000 P 297ad69a59194383bb21171ce9c5c479 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 297ad69a59194383bb21171ce9c5c479; no voters:
I20260807 07:02:51.753683 3205 leader_election.cc:288] T 00000000000000000000000000000000 P 297ad69a59194383bb21171ce9c5c479 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 07:02:51.753813 3212 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 297ad69a59194383bb21171ce9c5c479 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:02:51.754539 3212 raft_consensus.cc:686] T 00000000000000000000000000000000 P 297ad69a59194383bb21171ce9c5c479 [term 1 LEADER]: Becoming Leader. State: Replica: 297ad69a59194383bb21171ce9c5c479, State: Running, Role: LEADER
I20260807 07:02:51.755038 3212 consensus_queue.cc:237] T 00000000000000000000000000000000 P 297ad69a59194383bb21171ce9c5c479 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "297ad69a59194383bb21171ce9c5c479" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 36419 } }
I20260807 07:02:51.755554 3205 sys_catalog.cc:564] T 00000000000000000000000000000000 P 297ad69a59194383bb21171ce9c5c479 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:02:51.757493 3213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 297ad69a59194383bb21171ce9c5c479 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "297ad69a59194383bb21171ce9c5c479" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "297ad69a59194383bb21171ce9c5c479" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 36419 } } }
I20260807 07:02:51.757946 3213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 297ad69a59194383bb21171ce9c5c479 [sys.catalog]: This master's current role is: LEADER
I20260807 07:02:51.757784 3214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 297ad69a59194383bb21171ce9c5c479 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 297ad69a59194383bb21171ce9c5c479. Latest consensus state: current_term: 1 leader_uuid: "297ad69a59194383bb21171ce9c5c479" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "297ad69a59194383bb21171ce9c5c479" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 36419 } } }
I20260807 07:02:51.758209 3214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 297ad69a59194383bb21171ce9c5c479 [sys.catalog]: This master's current role is: LEADER
I20260807 07:02:51.759531 3222 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:02:51.763667 3222 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:02:51.771413 3222 catalog_manager.cc:1347] Generated new cluster ID: 80db6763a9f645259d7a8dd44e0ec808
I20260807 07:02:51.771512 3222 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:02:51.783951 3222 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:02:51.785868 3222 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:02:51.799409 3222 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 297ad69a59194383bb21171ce9c5c479: Generated new TSK 0
I20260807 07:02:51.800418 3222 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 07:02:51.815667 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.1:0
--local_ip_for_outbound_sockets=127.21.83.1
--webserver_interface=127.21.83.1
--webserver_port=0
--tserver_master_addrs=127.21.83.62:36419
--builtin_ntp_servers=127.21.83.20:35341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260807 07:02:52.143301 3240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:02:52.143579 3240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:02:52.143702 3240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:02:52.150698 3240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:02:52.150803 3240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:02:52.150938 3240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.1
I20260807 07:02:52.159624 3240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:35341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.83.1
--webserver_port=0
--tserver_master_addrs=127.21.83.62:36419
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.1
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:02:52.160794 3240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:02:52.162493 3240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:02:52.174294 3249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:52.174254 3250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:52.174399 3252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:02:52.175266 3240 server_base.cc:1034] running on GCE node
I20260807 07:02:52.175897 3240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:02:52.176733 3240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:02:52.177964 3240 hybrid_clock.cc:648] HybridClock initialized: now 1786086172177900 us; error 43 us; skew 500 ppm
I20260807 07:02:52.178288 3240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 07:02:52.180354 3240 webserver.cc:460] Webserver started at http://127.21.83.1:45963/ using document root <none> and password file <none>
I20260807 07:02:52.181049 3240 fs_manager.cc:359] Metadata directory not provided
I20260807 07:02:52.181155 3240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:02:52.181444 3240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:02:52.184211 3240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-0/data/instance:
uuid: "268e5c68a57249eb912f9303c36d4902"
format_stamp: "Formatted at 2026-08-07 07:02:52 on dist-test-slave-ncp9"
server_key: "447318acda892f4437d8756acf2f5825"
server_key_iv: "5832d35a8fbbf9f83c37adceb948185b"
server_key_version: "encryptionkey@0"
I20260807 07:02:52.184983 3240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-0/wal/instance:
uuid: "268e5c68a57249eb912f9303c36d4902"
format_stamp: "Formatted at 2026-08-07 07:02:52 on dist-test-slave-ncp9"
server_key: "447318acda892f4437d8756acf2f5825"
server_key_iv: "5832d35a8fbbf9f83c37adceb948185b"
server_key_version: "encryptionkey@0"
I20260807 07:02:52.189512 3240 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260807 07:02:52.192617 3262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:52.193913 3240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 07:02:52.194062 3240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-0/wal
uuid: "268e5c68a57249eb912f9303c36d4902"
format_stamp: "Formatted at 2026-08-07 07:02:52 on dist-test-slave-ncp9"
server_key: "447318acda892f4437d8756acf2f5825"
server_key_iv: "5832d35a8fbbf9f83c37adceb948185b"
server_key_version: "encryptionkey@0"
I20260807 07:02:52.194249 3240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:02:52.207175 3240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:02:52.207860 3240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:02:52.208094 3240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:02:52.208988 3240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:02:52.210253 3240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:02:52.210347 3240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:52.210477 3240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:02:52.210557 3240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:52.239674 3240 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.1:46685
I20260807 07:02:52.239727 3429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.1:46685 every 8 connection(s)
I20260807 07:02:52.240949 3240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
I20260807 07:02:52.244247 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 3240
I20260807 07:02:52.244467 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-0/wal/instance
I20260807 07:02:52.245288 21836 external_mini_cluster.cc:1428] Setting key 6e593286f0a3056e1df25f40e505720f
I20260807 07:02:52.247885 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.2:0
--local_ip_for_outbound_sockets=127.21.83.2
--webserver_interface=127.21.83.2
--webserver_port=0
--tserver_master_addrs=127.21.83.62:36419
--builtin_ntp_servers=127.21.83.20:35341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260807 07:02:52.253916 3209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:52.243489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:47083 (local address 127.21.83.62:36419)
0807 07:02:52.243672 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:52.243676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:52.245124 (+ 1448us) server_negotiation.cc:407] Connection header received
0807 07:02:52.246980 (+ 1856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:52.246984 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:52.247058 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:52.247155 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:52.248187 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:52.249171 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:52.250856 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:52.251377 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:52.251506 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:52.253126 (+ 1620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:52.253132 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:52.253136 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:02:52.253160 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:52.253200 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:02:52.253214 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:52.253299 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:52.253499 (+ 200us) server_negotiation.cc:299] Negotiation successful
0807 07:02:52.253604 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 07:02:52.254992 3433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:52.243967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:36419 (local address 127.21.83.1:47083)
0807 07:02:52.244976 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:52.245019 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:52.246731 (+ 1712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:52.247335 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:52.247342 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:52.247420 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:52.247963 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:52.247987 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:52.249306 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:52.249310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:52.250707 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:52.250721 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:52.251709 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:52.251715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:52.251934 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:52.252750 (+ 816us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 07:02:52.252753 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:52.252798 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 07:02:52.252886 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:52.253302 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:52.253309 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:52.253315 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 07:02:52.253472 (+ 157us) client_negotiation.cc:240] Negotiation successful
0807 07:02:52.253821 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":334,"threads_started":1}
I20260807 07:02:52.256680 3431 heartbeater.cc:347] Connected to a master server at 127.21.83.62:36419
I20260807 07:02:52.257079 3431 heartbeater.cc:464] Registering TS with master...
I20260807 07:02:52.258109 3431 heartbeater.cc:511] Master 127.21.83.62:36419 requested a full tablet report, sending...
I20260807 07:02:52.260496 3157 ts_manager.cc:194] Registered new tserver with Master: 268e5c68a57249eb912f9303c36d4902 (127.21.83.1:46685)
I20260807 07:02:52.262209 3157 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.83.1:47083
W20260807 07:02:52.579349 3434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:02:52.579650 3434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:02:52.579736 3434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:02:52.587146 3434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:02:52.587249 3434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:02:52.587353 3434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.2
I20260807 07:02:52.596329 3434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:35341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.83.2
--webserver_port=0
--tserver_master_addrs=127.21.83.62:36419
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.2
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:02:52.597579 3434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:02:52.599413 3434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:02:52.611694 3439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:52.611671 3440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:52.611790 3442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:02:52.612355 3434 server_base.cc:1034] running on GCE node
I20260807 07:02:52.612931 3434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:02:52.613821 3434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:02:52.615029 3434 hybrid_clock.cc:648] HybridClock initialized: now 1786086172614974 us; error 42 us; skew 500 ppm
I20260807 07:02:52.615324 3434 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 07:02:52.617499 3434 webserver.cc:460] Webserver started at http://127.21.83.2:39055/ using document root <none> and password file <none>
I20260807 07:02:52.618206 3434 fs_manager.cc:359] Metadata directory not provided
I20260807 07:02:52.618304 3434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:02:52.618611 3434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:02:52.621423 3434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-1/data/instance:
uuid: "ef700ed9f56142398781180ff308a571"
format_stamp: "Formatted at 2026-08-07 07:02:52 on dist-test-slave-ncp9"
server_key: "ceae69bdac4df3f11bcd53864c751338"
server_key_iv: "61a9e135a9909f1952e0ed9722631a4b"
server_key_version: "encryptionkey@0"
I20260807 07:02:52.622229 3434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-1/wal/instance:
uuid: "ef700ed9f56142398781180ff308a571"
format_stamp: "Formatted at 2026-08-07 07:02:52 on dist-test-slave-ncp9"
server_key: "ceae69bdac4df3f11bcd53864c751338"
server_key_iv: "61a9e135a9909f1952e0ed9722631a4b"
server_key_version: "encryptionkey@0"
I20260807 07:02:52.627364 3434 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 07:02:52.630992 3449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:52.632298 3434 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 07:02:52.632457 3434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-1/wal
uuid: "ef700ed9f56142398781180ff308a571"
format_stamp: "Formatted at 2026-08-07 07:02:52 on dist-test-slave-ncp9"
server_key: "ceae69bdac4df3f11bcd53864c751338"
server_key_iv: "61a9e135a9909f1952e0ed9722631a4b"
server_key_version: "encryptionkey@0"
I20260807 07:02:52.632665 3434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:02:52.650236 3434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:02:52.650990 3434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:02:52.651290 3434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:02:52.652209 3434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:02:52.653568 3434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:02:52.653656 3434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:52.653734 3434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:02:52.653795 3434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:52.684502 3434 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.2:44049
I20260807 07:02:52.684556 3616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.2:44049 every 8 connection(s)
I20260807 07:02:52.685839 3434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
I20260807 07:02:52.689618 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 3434
I20260807 07:02:52.689815 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-1/wal/instance
I20260807 07:02:52.690233 21836 external_mini_cluster.cc:1428] Setting key e48443978667d9db31e779ac665f3912
I20260807 07:02:52.693495 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.3:0
--local_ip_for_outbound_sockets=127.21.83.3
--webserver_interface=127.21.83.3
--webserver_port=0
--tserver_master_addrs=127.21.83.62:36419
--builtin_ntp_servers=127.21.83.20:35341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260807 07:02:52.699422 3209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:52.688285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:46969 (local address 127.21.83.62:36419)
0807 07:02:52.688449 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:52.688454 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:52.689801 (+ 1347us) server_negotiation.cc:407] Connection header received
0807 07:02:52.691905 (+ 2104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:52.691910 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:52.691985 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:52.692053 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:52.693663 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:52.694768 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:52.696802 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:52.697238 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:52.697326 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:52.698775 (+ 1449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:52.698779 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:52.698782 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:02:52.698799 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:52.698831 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:02:52.698842 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:52.698908 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:52.699115 (+ 207us) server_negotiation.cc:299] Negotiation successful
0807 07:02:52.699196 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 07:02:52.700651 3619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:52.688700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:36419 (local address 127.21.83.2:46969)
0807 07:02:52.689644 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:52.689688 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:52.691600 (+ 1912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:52.692572 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:52.692582 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:52.692702 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:52.693489 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:52.693516 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:52.694932 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:52.694937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:52.696676 (+ 1739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:52.696693 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:52.697456 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:52.697463 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:52.697728 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:52.698457 (+ 729us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 07:02:52.698461 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:52.698509 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 07:02:52.698600 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:52.698926 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:52.698936 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:52.698943 (+ 7us) client_negotiation.cc:769] Sending connection context
0807 07:02:52.699102 (+ 159us) client_negotiation.cc:240] Negotiation successful
0807 07:02:52.699450 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":329,"threads_started":1}
I20260807 07:02:52.701951 3617 heartbeater.cc:347] Connected to a master server at 127.21.83.62:36419
I20260807 07:02:52.702234 3617 heartbeater.cc:464] Registering TS with master...
I20260807 07:02:52.703073 3617 heartbeater.cc:511] Master 127.21.83.62:36419 requested a full tablet report, sending...
I20260807 07:02:52.704360 3157 ts_manager.cc:194] Registered new tserver with Master: ef700ed9f56142398781180ff308a571 (127.21.83.2:44049)
I20260807 07:02:52.705281 3157 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.83.2:46969
W20260807 07:02:53.034147 3620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:02:53.034478 3620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:02:53.034607 3620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:02:53.042172 3620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:02:53.042299 3620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:02:53.042446 3620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.3
I20260807 07:02:53.051584 3620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:35341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.83.3
--webserver_port=0
--tserver_master_addrs=127.21.83.62:36419
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.3
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:02:53.052861 3620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:02:53.054589 3620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:02:53.067729 3626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:53.067761 3630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:53.067816 3628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:02:53.068676 3620 server_base.cc:1034] running on GCE node
I20260807 07:02:53.069375 3620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:02:53.070422 3620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:02:53.071642 3620 hybrid_clock.cc:648] HybridClock initialized: now 1786086173071588 us; error 40 us; skew 500 ppm
I20260807 07:02:53.071974 3620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 07:02:53.074450 3620 webserver.cc:460] Webserver started at http://127.21.83.3:33763/ using document root <none> and password file <none>
I20260807 07:02:53.075191 3620 fs_manager.cc:359] Metadata directory not provided
I20260807 07:02:53.075300 3620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:02:53.075651 3620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:02:53.078780 3620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-2/data/instance:
uuid: "ff05504d3e244545b2e90d341c124521"
format_stamp: "Formatted at 2026-08-07 07:02:53 on dist-test-slave-ncp9"
server_key: "2b4efa8613e863f5d1245833933e1f1e"
server_key_iv: "1ae59db9c3f741dbd7cff8f537b2cbd8"
server_key_version: "encryptionkey@0"
I20260807 07:02:53.079607 3620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-2/wal/instance:
uuid: "ff05504d3e244545b2e90d341c124521"
format_stamp: "Formatted at 2026-08-07 07:02:53 on dist-test-slave-ncp9"
server_key: "2b4efa8613e863f5d1245833933e1f1e"
server_key_iv: "1ae59db9c3f741dbd7cff8f537b2cbd8"
server_key_version: "encryptionkey@0"
I20260807 07:02:53.084955 3620 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260807 07:02:53.088236 3641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:53.089459 3620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 07:02:53.089632 3620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-2/wal
uuid: "ff05504d3e244545b2e90d341c124521"
format_stamp: "Formatted at 2026-08-07 07:02:53 on dist-test-slave-ncp9"
server_key: "2b4efa8613e863f5d1245833933e1f1e"
server_key_iv: "1ae59db9c3f741dbd7cff8f537b2cbd8"
server_key_version: "encryptionkey@0"
I20260807 07:02:53.089838 3620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:02:53.106521 3620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:02:53.107270 3620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:02:53.107524 3620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:02:53.108423 3620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:02:53.109807 3620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:02:53.109905 3620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:53.110015 3620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:02:53.110086 3620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:53.142217 3620 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.3:42215
I20260807 07:02:53.142271 3804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.3:42215 every 8 connection(s)
I20260807 07:02:53.143478 3620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
I20260807 07:02:53.153196 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 3620
I20260807 07:02:53.153409 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-2/wal/instance
I20260807 07:02:53.153939 21836 external_mini_cluster.cc:1428] Setting key 0164d0ac39c249dffb0e7219b9143534
I20260807 07:02:53.154917 3209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:53.145943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:33561 (local address 127.21.83.62:36419)
0807 07:02:53.146144 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:53.146151 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:53.147456 (+ 1305us) server_negotiation.cc:407] Connection header received
0807 07:02:53.149123 (+ 1667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:53.149127 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:53.149193 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:53.149275 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:53.150170 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:53.150858 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:53.152287 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:53.152683 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:53.152765 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:53.154190 (+ 1425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:53.154195 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:53.154198 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:02:53.154215 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:53.154245 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:02:53.154257 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:53.154324 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:53.154552 (+ 228us) server_negotiation.cc:299] Negotiation successful
0807 07:02:53.154664 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 07:02:53.155907 3807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:53.146475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:36419 (local address 127.21.83.3:33561)
0807 07:02:53.147314 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:53.147348 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:53.148865 (+ 1517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:53.149447 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:53.149455 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:53.149529 (+ 74us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:53.150052 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:53.150066 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:53.150979 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:53.150982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:53.152174 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:53.152190 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:53.152874 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:53.152881 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:53.153180 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:53.153878 (+ 698us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 07:02:53.153882 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:53.153929 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 07:02:53.154018 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:53.154346 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:53.154356 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:53.154363 (+ 7us) client_negotiation.cc:769] Sending connection context
0807 07:02:53.154522 (+ 159us) client_negotiation.cc:240] Negotiation successful
0807 07:02:53.154843 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":264,"threads_started":1}
I20260807 07:02:53.157173 3805 heartbeater.cc:347] Connected to a master server at 127.21.83.62:36419
I20260807 07:02:53.157426 3805 heartbeater.cc:464] Registering TS with master...
I20260807 07:02:53.158100 3805 heartbeater.cc:511] Master 127.21.83.62:36419 requested a full tablet report, sending...
I20260807 07:02:53.159317 3156 ts_manager.cc:194] Registered new tserver with Master: ff05504d3e244545b2e90d341c124521 (127.21.83.3:42215)
I20260807 07:02:53.160171 3156 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.83.3:33561
I20260807 07:02:53.168675 21836 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260807 07:02:53.174361 3209 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:53.172951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38652 (local address 127.21.83.62:36419)
0807 07:02:53.173139 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:53.173145 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:53.173534 (+ 389us) server_negotiation.cc:407] Connection header received
0807 07:02:53.173644 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:53.173648 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:53.173725 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:53.173798 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:53.174132 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38652: BlockingRecv error: recv got EOF from 127.0.0.1:38652 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260807 07:02:53.175392 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 3240
I20260807 07:02:53.182691 3814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:53.176204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48780 (local address 127.21.83.1:46685)
0807 07:02:53.176673 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:53.176679 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:53.176702 (+ 23us) server_negotiation.cc:407] Connection header received
0807 07:02:53.176753 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:53.176758 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:53.176949 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:53.177080 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:53.177471 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:53.178402 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:53.179635 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:53.180159 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:53.180277 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:53.181310 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:53.181321 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:53.181334 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:02:53.181893 (+ 559us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:53.181937 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:02:53.181956 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:53.182060 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:53.182231 (+ 171us) server_negotiation.cc:299] Negotiation successful
0807 07:02:53.182355 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":208,"threads_started":1}
I20260807 07:02:53.183743 3420 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:02:53.265925 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 3240
I20260807 07:02:53.266032 3431 heartbeater.cc:503] Master 127.21.83.62:36419 was elected leader, sending a full tablet report...
I20260807 07:02:53.286430 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 3434
I20260807 07:02:53.294162 3817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:53.287329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50922 (local address 127.21.83.2:44049)
0807 07:02:53.287881 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:53.287888 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:53.287917 (+ 29us) server_negotiation.cc:407] Connection header received
0807 07:02:53.287983 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:53.287990 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:53.288213 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:53.288344 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:53.288709 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:53.289773 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:53.290947 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:53.291500 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:53.291592 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:53.292517 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:53.292526 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:53.292539 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:02:53.293331 (+ 792us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:53.293388 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:02:53.293410 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:53.293497 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:53.293705 (+ 208us) server_negotiation.cc:299] Negotiation successful
0807 07:02:53.293846 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":279,"threads_started":1}
I20260807 07:02:53.295459 3610 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:02:53.377971 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 3434
I20260807 07:02:53.401705 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 3620
I20260807 07:02:53.409404 3821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:53.402744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50488 (local address 127.21.83.3:42215)
0807 07:02:53.403399 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:53.403405 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:53.403434 (+ 29us) server_negotiation.cc:407] Connection header received
0807 07:02:53.403492 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:53.403498 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:53.403715 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:53.403833 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:53.404219 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:53.405290 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:53.406495 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:53.407046 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:53.407138 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:53.408023 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:53.408031 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:53.408043 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:02:53.408570 (+ 527us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:53.408614 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:02:53.408633 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:53.408726 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:53.408816 (+ 90us) server_negotiation.cc:299] Negotiation successful
0807 07:02:53.409042 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":345,"threads_started":1}
I20260807 07:02:53.410583 3798 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:02:53.495452 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 3620
I20260807 07:02:53.512033 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 3118
I20260807 07:02:53.513199 3199 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:02:53.615386 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 3118
2026-08-07T07:02:53Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2400 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-07T07:02:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:02:53Z Disabled control of system clock
I20260807 07:02:53.699126 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:39863
--webserver_interface=127.21.83.62
--webserver_port=0
--builtin_ntp_servers=127.21.83.20:43793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:39863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260807 07:02:54.021054 3828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:02:54.021318 3828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:02:54.021409 3828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:02:54.028364 3828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:02:54.028456 3828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:02:54.028507 3828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:02:54.028571 3828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:02:54.028613 3828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:02:54.037333 3828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:43793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:39863
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:39863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:02:54.038430 3828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:02:54.040094 3828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:02:54.050833 3838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:54.050907 3837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:54.050890 3842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:02:54.051584 3828 server_base.cc:1034] running on GCE node
I20260807 07:02:54.052310 3828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:02:54.053596 3828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:02:54.054787 3828 hybrid_clock.cc:648] HybridClock initialized: now 1786086174054739 us; error 39 us; skew 500 ppm
I20260807 07:02:54.055099 3828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 07:02:54.060352 3828 webserver.cc:460] Webserver started at http://127.21.83.62:36635/ using document root <none> and password file <none>
I20260807 07:02:54.061028 3828 fs_manager.cc:359] Metadata directory not provided
I20260807 07:02:54.061131 3828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:02:54.061422 3828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:02:54.064276 3828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "127bdd2068434223834778ef9301a3d6"
format_stamp: "Formatted at 2026-08-07 07:02:54 on dist-test-slave-ncp9"
server_key: "a8b952cf45c8e86ddaeffc8758a7c346"
server_key_iv: "a1f035e0f3d7807e459509665ede9657"
server_key_version: "encryptionkey@0"
I20260807 07:02:54.065192 3828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "127bdd2068434223834778ef9301a3d6"
format_stamp: "Formatted at 2026-08-07 07:02:54 on dist-test-slave-ncp9"
server_key: "a8b952cf45c8e86ddaeffc8758a7c346"
server_key_iv: "a1f035e0f3d7807e459509665ede9657"
server_key_version: "encryptionkey@0"
I20260807 07:02:54.070942 3828 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 07:02:54.074829 3850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:54.076296 3828 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 07:02:54.076489 3828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "127bdd2068434223834778ef9301a3d6"
format_stamp: "Formatted at 2026-08-07 07:02:54 on dist-test-slave-ncp9"
server_key: "a8b952cf45c8e86ddaeffc8758a7c346"
server_key_iv: "a1f035e0f3d7807e459509665ede9657"
server_key_version: "encryptionkey@0"
I20260807 07:02:54.076730 3828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:02:54.083076 3828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:02:54.083988 3828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:02:54.084300 3828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:02:54.102833 3828 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.62:39863
I20260807 07:02:54.102870 3926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.62:39863 every 8 connection(s)
I20260807 07:02:54.104137 3828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:02:54.107649 3927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:02:54.111975 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 3828
I20260807 07:02:54.112178 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:02:54.113044 21836 external_mini_cluster.cc:1428] Setting key 829378e56fe2c247f0c5d6ad728de96c
I20260807 07:02:54.119580 3927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 127bdd2068434223834778ef9301a3d6: Bootstrap starting.
I20260807 07:02:54.122784 3931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:54.115216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51976 (local address 127.21.83.62:39863)
0807 07:02:54.116395 (+ 1179us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:54.116417 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:54.116532 (+ 115us) server_negotiation.cc:407] Connection header received
0807 07:02:54.118887 (+ 2355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:54.118913 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:54.119417 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:54.119819 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:54.120798 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:54.120806 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:54.120820 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:02:54.120838 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:54.120938 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:02:54.120962 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:54.121196 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:54.121349 (+ 153us) server_negotiation.cc:299] Negotiation successful
0807 07:02:54.121619 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":418,"threads_started":1}
I20260807 07:02:54.123394 3927 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 127bdd2068434223834778ef9301a3d6: Neither blocks nor log segments found. Creating new log.
I20260807 07:02:54.124374 3927 log.cc:824] T 00000000000000000000000000000000 P 127bdd2068434223834778ef9301a3d6: Log is configured to *not* fsync() on all Append() calls
I20260807 07:02:54.127031 3927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 127bdd2068434223834778ef9301a3d6: No bootstrap required, opened a new log
I20260807 07:02:54.134268 3927 raft_consensus.cc:348] T 00000000000000000000000000000000 P 127bdd2068434223834778ef9301a3d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127bdd2068434223834778ef9301a3d6" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39863 } }
I20260807 07:02:54.134760 3927 raft_consensus.cc:374] T 00000000000000000000000000000000 P 127bdd2068434223834778ef9301a3d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:02:54.134872 3927 raft_consensus.cc:729] T 00000000000000000000000000000000 P 127bdd2068434223834778ef9301a3d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 127bdd2068434223834778ef9301a3d6, State: Initialized, Role: FOLLOWER
I20260807 07:02:54.135438 3927 consensus_queue.cc:260] T 00000000000000000000000000000000 P 127bdd2068434223834778ef9301a3d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127bdd2068434223834778ef9301a3d6" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39863 } }
I20260807 07:02:54.135668 3927 raft_consensus.cc:388] T 00000000000000000000000000000000 P 127bdd2068434223834778ef9301a3d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:02:54.135802 3927 raft_consensus.cc:482] T 00000000000000000000000000000000 P 127bdd2068434223834778ef9301a3d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:02:54.135957 3927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 127bdd2068434223834778ef9301a3d6 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:02:54.138362 3927 raft_consensus.cc:504] T 00000000000000000000000000000000 P 127bdd2068434223834778ef9301a3d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127bdd2068434223834778ef9301a3d6" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39863 } }
I20260807 07:02:54.138864 3927 leader_election.cc:302] T 00000000000000000000000000000000 P 127bdd2068434223834778ef9301a3d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 127bdd2068434223834778ef9301a3d6; no voters:
I20260807 07:02:54.139472 3927 leader_election.cc:288] T 00000000000000000000000000000000 P 127bdd2068434223834778ef9301a3d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 07:02:54.139597 3933 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 127bdd2068434223834778ef9301a3d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:02:54.140239 3933 raft_consensus.cc:686] T 00000000000000000000000000000000 P 127bdd2068434223834778ef9301a3d6 [term 1 LEADER]: Becoming Leader. State: Replica: 127bdd2068434223834778ef9301a3d6, State: Running, Role: LEADER
I20260807 07:02:54.140758 3933 consensus_queue.cc:237] T 00000000000000000000000000000000 P 127bdd2068434223834778ef9301a3d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127bdd2068434223834778ef9301a3d6" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39863 } }
I20260807 07:02:54.141554 3927 sys_catalog.cc:564] T 00000000000000000000000000000000 P 127bdd2068434223834778ef9301a3d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:02:54.143661 3934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 127bdd2068434223834778ef9301a3d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "127bdd2068434223834778ef9301a3d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127bdd2068434223834778ef9301a3d6" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39863 } } }
I20260807 07:02:54.143694 3936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 127bdd2068434223834778ef9301a3d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 127bdd2068434223834778ef9301a3d6. Latest consensus state: current_term: 1 leader_uuid: "127bdd2068434223834778ef9301a3d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127bdd2068434223834778ef9301a3d6" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39863 } } }
I20260807 07:02:54.144071 3934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 127bdd2068434223834778ef9301a3d6 [sys.catalog]: This master's current role is: LEADER
I20260807 07:02:54.144212 3936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 127bdd2068434223834778ef9301a3d6 [sys.catalog]: This master's current role is: LEADER
I20260807 07:02:54.145339 3939 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:02:54.150794 3939 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:02:54.159710 3939 catalog_manager.cc:1347] Generated new cluster ID: a7ec5a4a20da47cd8deac9e60964614d
I20260807 07:02:54.159823 3939 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:02:54.183600 3939 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:02:54.184043 3939 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:02:54.190847 3939 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 127bdd2068434223834778ef9301a3d6: Generated new TSK 0
I20260807 07:02:54.191553 3939 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 07:02:54.197911 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.1:0
--local_ip_for_outbound_sockets=127.21.83.1
--webserver_interface=127.21.83.1
--webserver_port=0
--tserver_master_addrs=127.21.83.62:39863
--builtin_ntp_servers=127.21.83.20:43793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260807 07:02:54.543864 3958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:02:54.544274 3958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:02:54.544430 3958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:02:54.554553 3958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:02:54.554674 3958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:02:54.554790 3958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.1
I20260807 07:02:54.564275 3958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:43793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.83.1
--webserver_port=0
--tserver_master_addrs=127.21.83.62:39863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.1
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:02:54.565614 3958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:02:54.567437 3958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:02:54.580662 3967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:54.580664 3964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:54.580981 3971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:02:54.581516 3958 server_base.cc:1034] running on GCE node
I20260807 07:02:54.582283 3958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:02:54.583273 3958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:02:54.584513 3958 hybrid_clock.cc:648] HybridClock initialized: now 1786086174584448 us; error 50 us; skew 500 ppm
I20260807 07:02:54.584916 3958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 07:02:54.587224 3958 webserver.cc:460] Webserver started at http://127.21.83.1:33291/ using document root <none> and password file <none>
I20260807 07:02:54.587980 3958 fs_manager.cc:359] Metadata directory not provided
I20260807 07:02:54.588088 3958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:02:54.588404 3958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:02:54.594233 3958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-0/data/instance:
uuid: "bedb752f7d074e99b502992e9bef4303"
format_stamp: "Formatted at 2026-08-07 07:02:54 on dist-test-slave-ncp9"
server_key: "645845e75f3a1a724cb81a244554335f"
server_key_iv: "e86251c43c69310a88c300b2a17ff1a5"
server_key_version: "encryptionkey@0"
I20260807 07:02:54.595088 3958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-0/wal/instance:
uuid: "bedb752f7d074e99b502992e9bef4303"
format_stamp: "Formatted at 2026-08-07 07:02:54 on dist-test-slave-ncp9"
server_key: "645845e75f3a1a724cb81a244554335f"
server_key_iv: "e86251c43c69310a88c300b2a17ff1a5"
server_key_version: "encryptionkey@0"
I20260807 07:02:54.600559 3958 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 07:02:54.604056 3978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:54.605370 3958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 07:02:54.605533 3958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-0/wal
uuid: "bedb752f7d074e99b502992e9bef4303"
format_stamp: "Formatted at 2026-08-07 07:02:54 on dist-test-slave-ncp9"
server_key: "645845e75f3a1a724cb81a244554335f"
server_key_iv: "e86251c43c69310a88c300b2a17ff1a5"
server_key_version: "encryptionkey@0"
I20260807 07:02:54.605749 3958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:02:54.611075 3958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:02:54.611847 3958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:02:54.612092 3958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:02:54.613042 3958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:02:54.614426 3958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:02:54.614522 3958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:54.614634 3958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:02:54.614704 3958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:54.644798 3958 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.1:45703
I20260807 07:02:54.644827 4142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.1:45703 every 8 connection(s)
I20260807 07:02:54.646078 3958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
I20260807 07:02:54.652976 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 3958
I20260807 07:02:54.653174 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-0/wal/instance
I20260807 07:02:54.653795 21836 external_mini_cluster.cc:1428] Setting key 4e726fcd751030586692300e6f7e1975
I20260807 07:02:54.654135 4148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:54.648625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:37187 (local address 127.21.83.62:39863)
0807 07:02:54.649064 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:54.649069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:54.650146 (+ 1077us) server_negotiation.cc:407] Connection header received
0807 07:02:54.651886 (+ 1740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:54.651892 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:54.651957 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:54.652053 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:54.653419 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:54.653424 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:54.653428 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:02:54.653444 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:54.653486 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:02:54.653498 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:54.653600 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:54.653787 (+ 187us) server_negotiation.cc:299] Negotiation successful
0807 07:02:54.653853 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":215,"threads_started":1}
I20260807 07:02:54.655088 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:54.649130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39863 (local address 127.21.83.1:37187)
0807 07:02:54.649997 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:54.650039 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:54.651618 (+ 1579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:54.652300 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:54.652307 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:54.652390 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:54.653092 (+ 702us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 07:02:54.653095 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:54.653149 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 07:02:54.653243 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:54.653584 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:54.653595 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:54.653602 (+ 7us) client_negotiation.cc:769] Sending connection context
0807 07:02:54.653853 (+ 251us) client_negotiation.cc:240] Negotiation successful
0807 07:02:54.654086 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":270,"threads_started":1}
I20260807 07:02:54.656425 4146 heartbeater.cc:347] Connected to a master server at 127.21.83.62:39863
I20260807 07:02:54.656433 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.2:0
--local_ip_for_outbound_sockets=127.21.83.2
--webserver_interface=127.21.83.2
--webserver_port=0
--tserver_master_addrs=127.21.83.62:39863
--builtin_ntp_servers=127.21.83.20:43793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260807 07:02:54.656709 4146 heartbeater.cc:464] Registering TS with master...
I20260807 07:02:54.657299 4146 heartbeater.cc:511] Master 127.21.83.62:39863 requested a full tablet report, sending...
I20260807 07:02:54.659453 3877 ts_manager.cc:194] Registered new tserver with Master: bedb752f7d074e99b502992e9bef4303 (127.21.83.1:45703)
W20260807 07:02:54.985211 4152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:02:54.985507 4152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:02:54.985606 4152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:02:54.993455 4152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:02:54.993582 4152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:02:54.993705 4152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.2
I20260807 07:02:55.003152 4152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:43793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.83.2
--webserver_port=0
--tserver_master_addrs=127.21.83.62:39863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.2
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:02:55.004422 4152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:02:55.006245 4152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:02:55.018877 4161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:55.018818 4165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:55.018919 4163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:02:55.020084 4152 server_base.cc:1034] running on GCE node
I20260807 07:02:55.020640 4152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:02:55.021526 4152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:02:55.022727 4152 hybrid_clock.cc:648] HybridClock initialized: now 1786086175022686 us; error 34 us; skew 500 ppm
I20260807 07:02:55.023015 4152 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 07:02:55.025219 4152 webserver.cc:460] Webserver started at http://127.21.83.2:40375/ using document root <none> and password file <none>
I20260807 07:02:55.025877 4152 fs_manager.cc:359] Metadata directory not provided
I20260807 07:02:55.025970 4152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:02:55.026278 4152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:02:55.029037 4152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-1/data/instance:
uuid: "ac1430de30f54ac188036907a44823ef"
format_stamp: "Formatted at 2026-08-07 07:02:55 on dist-test-slave-ncp9"
server_key: "f841be9def7b2c5c676f4954e48aef9c"
server_key_iv: "b9c77a280cdb11d22992c6dde1bac112"
server_key_version: "encryptionkey@0"
I20260807 07:02:55.029815 4152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-1/wal/instance:
uuid: "ac1430de30f54ac188036907a44823ef"
format_stamp: "Formatted at 2026-08-07 07:02:55 on dist-test-slave-ncp9"
server_key: "f841be9def7b2c5c676f4954e48aef9c"
server_key_iv: "b9c77a280cdb11d22992c6dde1bac112"
server_key_version: "encryptionkey@0"
I20260807 07:02:55.034713 4152 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260807 07:02:55.038196 4173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:55.039503 4152 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 07:02:55.039669 4152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-1/wal
uuid: "ac1430de30f54ac188036907a44823ef"
format_stamp: "Formatted at 2026-08-07 07:02:55 on dist-test-slave-ncp9"
server_key: "f841be9def7b2c5c676f4954e48aef9c"
server_key_iv: "b9c77a280cdb11d22992c6dde1bac112"
server_key_version: "encryptionkey@0"
I20260807 07:02:55.039883 4152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:02:55.045641 4152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:02:55.046428 4152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:02:55.046691 4152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:02:55.047675 4152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:02:55.049109 4152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:02:55.049208 4152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:55.049314 4152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:02:55.049392 4152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:55.079357 4152 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.2:32787
I20260807 07:02:55.079411 4339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.2:32787 every 8 connection(s)
I20260807 07:02:55.080627 4152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
I20260807 07:02:55.087045 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 4152
I20260807 07:02:55.087204 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-1/wal/instance
I20260807 07:02:55.087656 21836 external_mini_cluster.cc:1428] Setting key d26b94b7c55106764d45637ecea0c5b6
I20260807 07:02:55.088388 4148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:55.083085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:53901 (local address 127.21.83.62:39863)
0807 07:02:55.083264 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:55.083269 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:55.084506 (+ 1237us) server_negotiation.cc:407] Connection header received
0807 07:02:55.086184 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:55.086190 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:55.086253 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:55.086342 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:55.087658 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:55.087663 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:55.087666 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:02:55.087682 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:55.087715 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:02:55.087728 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:55.087840 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:55.088051 (+ 211us) server_negotiation.cc:299] Negotiation successful
0807 07:02:55.088101 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 07:02:55.089318 4344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:55.083520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39863 (local address 127.21.83.2:53901)
0807 07:02:55.084357 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:55.084397 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:55.085933 (+ 1536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:55.086572 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:55.086580 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:55.086679 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:55.087375 (+ 696us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 07:02:55.087378 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:55.087412 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 07:02:55.087491 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:55.087852 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:55.087864 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:55.087872 (+ 8us) client_negotiation.cc:769] Sending connection context
0807 07:02:55.088140 (+ 268us) client_negotiation.cc:240] Negotiation successful
0807 07:02:55.088376 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":255,"threads_started":1}
I20260807 07:02:55.090262 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.3:0
--local_ip_for_outbound_sockets=127.21.83.3
--webserver_interface=127.21.83.3
--webserver_port=0
--tserver_master_addrs=127.21.83.62:39863
--builtin_ntp_servers=127.21.83.20:43793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260807 07:02:55.091481 4340 heartbeater.cc:347] Connected to a master server at 127.21.83.62:39863
I20260807 07:02:55.091740 4340 heartbeater.cc:464] Registering TS with master...
I20260807 07:02:55.092254 4340 heartbeater.cc:511] Master 127.21.83.62:39863 requested a full tablet report, sending...
I20260807 07:02:55.093554 3877 ts_manager.cc:194] Registered new tserver with Master: ac1430de30f54ac188036907a44823ef (127.21.83.2:32787)
W20260807 07:02:55.420367 4345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:02:55.420611 4345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:02:55.420724 4345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:02:55.427809 4345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:02:55.427901 4345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:02:55.428050 4345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.3
I20260807 07:02:55.436753 4345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:43793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.83.3
--webserver_port=0
--tserver_master_addrs=127.21.83.62:39863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.3
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:02:55.437935 4345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:02:55.439545 4345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:02:55.454026 4354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:55.454048 4355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:55.454375 4359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:02:55.454849 4345 server_base.cc:1034] running on GCE node
I20260807 07:02:55.455384 4345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:02:55.456198 4345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:02:55.457387 4345 hybrid_clock.cc:648] HybridClock initialized: now 1786086175457326 us; error 46 us; skew 500 ppm
I20260807 07:02:55.457679 4345 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 07:02:55.462450 4345 webserver.cc:460] Webserver started at http://127.21.83.3:32921/ using document root <none> and password file <none>
I20260807 07:02:55.463054 4345 fs_manager.cc:359] Metadata directory not provided
I20260807 07:02:55.463156 4345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:02:55.463451 4345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:02:55.466090 4345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-2/data/instance:
uuid: "03c7413c77534609bef9d29130f5a9a1"
format_stamp: "Formatted at 2026-08-07 07:02:55 on dist-test-slave-ncp9"
server_key: "6ce47fd2d7daf54b8121e4da387e59a1"
server_key_iv: "0d463c8de0b3dba1d32abade50058a9d"
server_key_version: "encryptionkey@0"
I20260807 07:02:55.466818 4345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-2/wal/instance:
uuid: "03c7413c77534609bef9d29130f5a9a1"
format_stamp: "Formatted at 2026-08-07 07:02:55 on dist-test-slave-ncp9"
server_key: "6ce47fd2d7daf54b8121e4da387e59a1"
server_key_iv: "0d463c8de0b3dba1d32abade50058a9d"
server_key_version: "encryptionkey@0"
I20260807 07:02:55.471544 4345 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 07:02:55.474653 4370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:55.475788 4345 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260807 07:02:55.475934 4345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-2/wal
uuid: "03c7413c77534609bef9d29130f5a9a1"
format_stamp: "Formatted at 2026-08-07 07:02:55 on dist-test-slave-ncp9"
server_key: "6ce47fd2d7daf54b8121e4da387e59a1"
server_key_iv: "0d463c8de0b3dba1d32abade50058a9d"
server_key_version: "encryptionkey@0"
I20260807 07:02:55.476135 4345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:02:55.481381 4345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:02:55.482092 4345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:02:55.482344 4345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:02:55.483176 4345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:02:55.484436 4345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:02:55.484530 4345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:55.484639 4345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:02:55.484715 4345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:55.515451 4345 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.3:41477
I20260807 07:02:55.515517 4527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.3:41477 every 8 connection(s)
I20260807 07:02:55.516728 4345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
I20260807 07:02:55.518653 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 4345
I20260807 07:02:55.518836 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786086078080656-21836-0/minicluster-data/ts-2/wal/instance
I20260807 07:02:55.519389 21836 external_mini_cluster.cc:1428] Setting key 46ce55f8fdf0df61ab0bcef01254738b
I20260807 07:02:55.524587 4148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:55.519251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:40949 (local address 127.21.83.62:39863)
0807 07:02:55.519415 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:55.519419 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:55.521101 (+ 1682us) server_negotiation.cc:407] Connection header received
0807 07:02:55.522855 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:55.522859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:55.522911 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:55.522992 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:55.524063 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:55.524067 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:55.524071 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:02:55.524085 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:55.524115 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:02:55.524126 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:55.524222 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:55.524340 (+ 118us) server_negotiation.cc:299] Negotiation successful
0807 07:02:55.524388 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260807 07:02:55.525447 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:55.519867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39863 (local address 127.21.83.3:40949)
0807 07:02:55.520935 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:55.520984 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:55.522631 (+ 1647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:55.523164 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:55.523171 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:55.523252 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:55.523815 (+ 563us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 07:02:55.523817 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:55.523851 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 07:02:55.523929 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:55.524199 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:55.524204 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:55.524209 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 07:02:55.524398 (+ 189us) client_negotiation.cc:240] Negotiation successful
0807 07:02:55.524579 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":329,"threads_started":1}
I20260807 07:02:55.526629 4529 heartbeater.cc:347] Connected to a master server at 127.21.83.62:39863
I20260807 07:02:55.526901 4529 heartbeater.cc:464] Registering TS with master...
I20260807 07:02:55.527426 4529 heartbeater.cc:511] Master 127.21.83.62:39863 requested a full tablet report, sending...
I20260807 07:02:55.528618 3877 ts_manager.cc:194] Registered new tserver with Master: 03c7413c77534609bef9d29130f5a9a1 (127.21.83.3:41477)
I20260807 07:02:55.533979 21836 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260807 07:02:55.539574 4148 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:02:55.538120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51992 (local address 127.21.83.62:39863)
0807 07:02:55.538280 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:55.538286 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:55.538783 (+ 497us) server_negotiation.cc:407] Connection header received
0807 07:02:55.538898 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:55.538902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:55.538947 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:55.539020 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:55.539352 (+ 332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51992: BlockingRecv error: recv got EOF from 127.0.0.1:51992 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260807 07:02:55.540390 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 3958
I20260807 07:02:55.543848 4543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:55.541312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60414 (local address 127.21.83.1:45703)
0807 07:02:55.541878 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:55.541884 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:55.541909 (+ 25us) server_negotiation.cc:407] Connection header received
0807 07:02:55.541962 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:55.541967 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:55.542175 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:55.542309 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:55.543098 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:55.543104 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:55.543114 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:02:55.543132 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:55.543180 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:02:55.543215 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:55.543330 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:55.543529 (+ 199us) server_negotiation.cc:299] Negotiation successful
0807 07:02:55.543620 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":289,"threads_started":1}
I20260807 07:02:55.545074 4137 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:02:55.624147 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 3958
I20260807 07:02:55.648831 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 4152
I20260807 07:02:55.651953 4545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:55.649741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55370 (local address 127.21.83.2:32787)
0807 07:02:55.650192 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:55.650198 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:55.650223 (+ 25us) server_negotiation.cc:407] Connection header received
0807 07:02:55.650281 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:55.650287 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:55.650484 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:55.650608 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:55.651411 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:55.651417 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:55.651427 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:02:55.651444 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:55.651482 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:02:55.651495 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:55.651602 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:55.651650 (+ 48us) server_negotiation.cc:299] Negotiation successful
0807 07:02:55.651725 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":196,"threads_started":1}
I20260807 07:02:55.653107 4329 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:02:55.732869 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 4152
I20260807 07:02:55.753412 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 4345
I20260807 07:02:55.757457 4547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:55.754331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41914 (local address 127.21.83.3:41477)
0807 07:02:55.755009 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:55.755017 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:55.755049 (+ 32us) server_negotiation.cc:407] Connection header received
0807 07:02:55.755115 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:55.755122 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:55.755342 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:55.755490 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:55.756664 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:55.756671 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:55.756682 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:02:55.756700 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:55.756742 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:02:55.756757 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:55.756939 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:55.757029 (+ 90us) server_negotiation.cc:299] Negotiation successful
0807 07:02:55.757137 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":335,"threads_started":1}
I20260807 07:02:55.758773 4520 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:02:55.840754 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 4345
I20260807 07:02:55.862351 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 3828
I20260807 07:02:55.863826 3919 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:02:55.964933 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 3828
2026-08-07T07:02:55Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2351 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 07:02:56 dist-test-slave-ncp9 krb5kdc[4553](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 07:02:56 dist-test-slave-ncp9 krb5kdc[4553](info): set up 2 sockets
Aug 07 07:02:56 dist-test-slave-ncp9 krb5kdc[4553](info): commencing operation
krb5kdc: starting...
W20260807 07:02:57.164841 21836 mini_kdc.cc:121] Time spent starting KDC: real 1.153s user 0.002s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Aug 07 07:02:57 dist-test-slave-ncp9 krb5kdc[4553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T07:02:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:02:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
I20260807 07:02:57.436829 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:35031
--webserver_interface=127.21.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:35061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:35031
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:02:57.757927 4573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:02:57.758175 4573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:02:57.758271 4573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:02:57.765187 4573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:02:57.765276 4573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:02:57.765328 4573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:02:57.765381 4573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:02:57.765424 4573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:02:57.775024 4573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:35061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:35031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:35031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:02:57.776423 4573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:02:57.777770 4573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:02:57.790664 4581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:57.790694 4587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:57.790627 4580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:02:57.791863 4573 server_base.cc:1034] running on GCE node
I20260807 07:02:57.792659 4573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:02:57.794097 4573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:02:57.795295 4573 hybrid_clock.cc:648] HybridClock initialized: now 1786086177795254 us; error 34 us; skew 500 ppm
I20260807 07:02:57.795605 4573 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:02:57 dist-test-slave-ncp9 krb5kdc[4553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086177, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:02:57.798535 4573 init.cc:377] Logged in from keytab as kudu/127.21.83.62@KRBTEST.COM (short username kudu)
I20260807 07:02:57.799916 4573 webserver.cc:460] Webserver started at http://127.21.83.62:37409/ using document root <none> and password file <none>
I20260807 07:02:57.800531 4573 fs_manager.cc:359] Metadata directory not provided
I20260807 07:02:57.800637 4573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:02:57.800875 4573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:02:57.803740 4573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "29861f3d836b46ef9c50fbd3dee81a9f"
format_stamp: "Formatted at 2026-08-07 07:02:57 on dist-test-slave-ncp9"
server_key: "b91e946f2474d501331c8b8931f407d3"
server_key_iv: "b9673fa6a3c5c2a3886ae53febe0a7a3"
server_key_version: "encryptionkey@0"
I20260807 07:02:57.804421 4573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "29861f3d836b46ef9c50fbd3dee81a9f"
format_stamp: "Formatted at 2026-08-07 07:02:57 on dist-test-slave-ncp9"
server_key: "b91e946f2474d501331c8b8931f407d3"
server_key_iv: "b9673fa6a3c5c2a3886ae53febe0a7a3"
server_key_version: "encryptionkey@0"
I20260807 07:02:57.809185 4573 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.005s
I20260807 07:02:57.812299 4596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:57.813520 4573 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 07:02:57.813660 4573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "29861f3d836b46ef9c50fbd3dee81a9f"
format_stamp: "Formatted at 2026-08-07 07:02:57 on dist-test-slave-ncp9"
server_key: "b91e946f2474d501331c8b8931f407d3"
server_key_iv: "b9673fa6a3c5c2a3886ae53febe0a7a3"
server_key_version: "encryptionkey@0"
I20260807 07:02:57.813807 4573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:02:57.833366 4573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:02:57.837534 4573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:02:57.837801 4573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:02:57.855474 4573 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.62:35031
I20260807 07:02:57.855520 4662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.62:35031 every 8 connection(s)
I20260807 07:02:57.856773 4573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:02:57.860247 4663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:02:57.862569 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 4573
I20260807 07:02:57.862763 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:02:57.863425 21836 external_mini_cluster.cc:1428] Setting key 9334be450e5eff2b1936a1a31bde2df9
I20260807 07:02:57.873250 4663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29861f3d836b46ef9c50fbd3dee81a9f: Bootstrap starting.
I20260807 07:02:57.877233 4663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29861f3d836b46ef9c50fbd3dee81a9f: Neither blocks nor log segments found. Creating new log.
I20260807 07:02:57.878566 4663 log.cc:824] T 00000000000000000000000000000000 P 29861f3d836b46ef9c50fbd3dee81a9f: Log is configured to *not* fsync() on all Append() calls
Aug 07 07:02:57 dist-test-slave-ncp9 krb5kdc[4553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:02:57.881999 4663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29861f3d836b46ef9c50fbd3dee81a9f: No bootstrap required, opened a new log
I20260807 07:02:57.889160 4668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:57.865794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39900 (local address 127.21.83.62:35031)
0807 07:02:57.866643 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:57.866657 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:57.866702 (+ 45us) server_negotiation.cc:407] Connection header received
0807 07:02:57.868782 (+ 2080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:57.868812 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:57.869391 (+ 579us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:57.869782 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:57.871708 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:57.873339 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:57.875184 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:57.875914 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:57.876066 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:57.880926 (+ 4860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:57.880939 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:57.880957 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:57.881016 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:57.884855 (+ 3839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:57.885511 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:57.885521 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:57.885529 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:57.885740 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:57.886063 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:57.886068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:57.886071 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:57.886770 (+ 699us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:57.887072 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:57.887354 (+ 282us) server_negotiation.cc:299] Negotiation successful
0807 07:02:57.887831 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":386,"threads_started":1}
I20260807 07:02:57.892221 4663 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29861f3d836b46ef9c50fbd3dee81a9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29861f3d836b46ef9c50fbd3dee81a9f" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 35031 } }
I20260807 07:02:57.892539 4663 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29861f3d836b46ef9c50fbd3dee81a9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:02:57.892612 4663 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29861f3d836b46ef9c50fbd3dee81a9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29861f3d836b46ef9c50fbd3dee81a9f, State: Initialized, Role: FOLLOWER
I20260807 07:02:57.893110 4663 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29861f3d836b46ef9c50fbd3dee81a9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29861f3d836b46ef9c50fbd3dee81a9f" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 35031 } }
I20260807 07:02:57.893321 4663 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29861f3d836b46ef9c50fbd3dee81a9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:02:57.893442 4663 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29861f3d836b46ef9c50fbd3dee81a9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:02:57.893605 4663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29861f3d836b46ef9c50fbd3dee81a9f [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:02:57.895835 4663 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29861f3d836b46ef9c50fbd3dee81a9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29861f3d836b46ef9c50fbd3dee81a9f" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 35031 } }
I20260807 07:02:57.896268 4663 leader_election.cc:302] T 00000000000000000000000000000000 P 29861f3d836b46ef9c50fbd3dee81a9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29861f3d836b46ef9c50fbd3dee81a9f; no voters:
I20260807 07:02:57.896796 4663 leader_election.cc:288] T 00000000000000000000000000000000 P 29861f3d836b46ef9c50fbd3dee81a9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 07:02:57.896946 4670 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29861f3d836b46ef9c50fbd3dee81a9f [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:02:57.897605 4670 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29861f3d836b46ef9c50fbd3dee81a9f [term 1 LEADER]: Becoming Leader. State: Replica: 29861f3d836b46ef9c50fbd3dee81a9f, State: Running, Role: LEADER
I20260807 07:02:57.898118 4670 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29861f3d836b46ef9c50fbd3dee81a9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29861f3d836b46ef9c50fbd3dee81a9f" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 35031 } }
I20260807 07:02:57.898631 4663 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29861f3d836b46ef9c50fbd3dee81a9f [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:02:57.900509 4671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29861f3d836b46ef9c50fbd3dee81a9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29861f3d836b46ef9c50fbd3dee81a9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29861f3d836b46ef9c50fbd3dee81a9f" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 35031 } } }
I20260807 07:02:57.900885 4671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29861f3d836b46ef9c50fbd3dee81a9f [sys.catalog]: This master's current role is: LEADER
I20260807 07:02:57.901049 4673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29861f3d836b46ef9c50fbd3dee81a9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29861f3d836b46ef9c50fbd3dee81a9f. Latest consensus state: current_term: 1 leader_uuid: "29861f3d836b46ef9c50fbd3dee81a9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29861f3d836b46ef9c50fbd3dee81a9f" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 35031 } } }
I20260807 07:02:57.901747 4673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29861f3d836b46ef9c50fbd3dee81a9f [sys.catalog]: This master's current role is: LEADER
I20260807 07:02:57.902050 4678 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:02:57.907336 4678 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:02:57.915563 4678 catalog_manager.cc:1347] Generated new cluster ID: cdc8a895261c4f9b99a8b15332e263c4
I20260807 07:02:57.915683 4678 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:02:57.927587 4678 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:02:57.929080 4678 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:02:57.936117 4678 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29861f3d836b46ef9c50fbd3dee81a9f: Generated new TSK 0
I20260807 07:02:57.936957 4678 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.83.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab.
Entry for principal kudu/127.21.83.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab.
I20260807 07:02:58.010018 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.1:0
--local_ip_for_outbound_sockets=127.21.83.1
--webserver_interface=127.21.83.1
--webserver_port=0
--tserver_master_addrs=127.21.83.62:35031
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab
--principal=kudu/127.21.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:35061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:02:58.329120 4698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:02:58.329375 4698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:02:58.329473 4698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:02:58.336360 4698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:02:58.336457 4698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:02:58.336577 4698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.1
I20260807 07:02:58.345811 4698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:35061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab
--principal=kudu/127.21.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.83.1
--webserver_port=0
--tserver_master_addrs=127.21.83.62:35031
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.1
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:02:58.346944 4698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:02:58.348110 4698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:02:58.360431 4703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:58.360404 4704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:58.360469 4710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:02:58.361258 4698 server_base.cc:1034] running on GCE node
I20260807 07:02:58.361805 4698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:02:58.362682 4698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:02:58.363914 4698 hybrid_clock.cc:648] HybridClock initialized: now 1786086178363860 us; error 42 us; skew 500 ppm
I20260807 07:02:58.364216 4698 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:02:58 dist-test-slave-ncp9 krb5kdc[4553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086178, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:02:58.367201 4698 init.cc:377] Logged in from keytab as kudu/127.21.83.1@KRBTEST.COM (short username kudu)
I20260807 07:02:58.368542 4698 webserver.cc:460] Webserver started at http://127.21.83.1:40519/ using document root <none> and password file <none>
I20260807 07:02:58.369215 4698 fs_manager.cc:359] Metadata directory not provided
I20260807 07:02:58.369329 4698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:02:58.369617 4698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:02:58.372367 4698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-0/data/instance:
uuid: "ee846b8d4ab94ef8bf6583c398214b8b"
format_stamp: "Formatted at 2026-08-07 07:02:58 on dist-test-slave-ncp9"
server_key: "0b787ada598ff6522e90546d49f54757"
server_key_iv: "569ea02c7dcff23a43c06285ffe09ff2"
server_key_version: "encryptionkey@0"
I20260807 07:02:58.373126 4698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-0/wal/instance:
uuid: "ee846b8d4ab94ef8bf6583c398214b8b"
format_stamp: "Formatted at 2026-08-07 07:02:58 on dist-test-slave-ncp9"
server_key: "0b787ada598ff6522e90546d49f54757"
server_key_iv: "569ea02c7dcff23a43c06285ffe09ff2"
server_key_version: "encryptionkey@0"
I20260807 07:02:58.377815 4698 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260807 07:02:58.380949 4719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:58.382098 4698 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260807 07:02:58.382261 4698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-0/wal
uuid: "ee846b8d4ab94ef8bf6583c398214b8b"
format_stamp: "Formatted at 2026-08-07 07:02:58 on dist-test-slave-ncp9"
server_key: "0b787ada598ff6522e90546d49f54757"
server_key_iv: "569ea02c7dcff23a43c06285ffe09ff2"
server_key_version: "encryptionkey@0"
I20260807 07:02:58.382431 4698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:02:58.398401 4698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:02:58.402321 4698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:02:58.402601 4698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:02:58.403414 4698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:02:58.404676 4698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:02:58.404745 4698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:58.404861 4698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:02:58.404950 4698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260807 07:02:58.432014 4698 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.1:38597
I20260807 07:02:58.432070 4866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.1:38597 every 8 connection(s)
I20260807 07:02:58.433174 4698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
I20260807 07:02:58.435603 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 4698
I20260807 07:02:58.435797 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-0/wal/instance
I20260807 07:02:58.436627 21836 external_mini_cluster.cc:1428] Setting key 215250f073a5dc7804ba7e4763df6d7d
Aug 07 07:02:58 dist-test-slave-ncp9 krb5kdc[4553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086178, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.1@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:02:58.452159 4872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:58.435451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:43727 (local address 127.21.83.62:35031)
0807 07:02:58.435869 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:58.435875 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:58.436951 (+ 1076us) server_negotiation.cc:407] Connection header received
0807 07:02:58.438952 (+ 2001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:58.438958 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:58.439031 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:58.439123 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:58.440570 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:58.441308 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:58.442962 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:58.443351 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:58.443461 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:58.446912 (+ 3451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:58.446917 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:58.446920 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:58.446951 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:58.449275 (+ 2324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:58.450019 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:58.450022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:58.450025 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:58.450132 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:58.450522 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:58.450524 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:58.450526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:58.450866 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:58.451011 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:58.451671 (+ 660us) server_negotiation.cc:299] Negotiation successful
0807 07:02:58.451883 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":386,"threads_started":1}
I20260807 07:02:58.453459 4873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:58.436054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:35031 (local address 127.21.83.1:43727)
0807 07:02:58.436802 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:58.436841 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:58.438716 (+ 1875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:58.439355 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:58.439363 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:58.439858 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:58.440411 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:58.440427 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:58.441450 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:58.441454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:58.442822 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:58.442835 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:58.443566 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:58.443574 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:58.443775 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:58.444348 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:58.444375 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:58.446711 (+ 2336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:58.449399 (+ 2688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:58.449407 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:58.449419 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:58.449929 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:58.450217 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:58.450221 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:58.450224 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:58.450437 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:58.451016 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:58.451022 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:58.451317 (+ 295us) client_negotiation.cc:769] Sending connection context
0807 07:02:58.451646 (+ 329us) client_negotiation.cc:240] Negotiation successful
0807 07:02:58.452141 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":304,"threads_started":1}
I20260807 07:02:58.455089 4867 heartbeater.cc:347] Connected to a master server at 127.21.83.62:35031
I20260807 07:02:58.455379 4867 heartbeater.cc:464] Registering TS with master...
I20260807 07:02:58.456135 4867 heartbeater.cc:511] Master 127.21.83.62:35031 requested a full tablet report, sending...
I20260807 07:02:58.458436 4621 ts_manager.cc:194] Registered new tserver with Master: ee846b8d4ab94ef8bf6583c398214b8b (127.21.83.1:38597)
I20260807 07:02:58.460402 4621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.1@KRBTEST.COM'} at 127.21.83.1:43727
WARNING: no policy specified for kudu/127.21.83.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab.
Entry for principal kudu/127.21.83.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab.
I20260807 07:02:58.497462 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.2:0
--local_ip_for_outbound_sockets=127.21.83.2
--webserver_interface=127.21.83.2
--webserver_port=0
--tserver_master_addrs=127.21.83.62:35031
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab
--principal=kudu/127.21.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:35061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:02:58.813566 4876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:02:58.813843 4876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:02:58.813948 4876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:02:58.821138 4876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:02:58.821231 4876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:02:58.821346 4876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.2
I20260807 07:02:58.830488 4876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:35061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab
--principal=kudu/127.21.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.83.2
--webserver_port=0
--tserver_master_addrs=127.21.83.62:35031
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.2
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:02:58.831550 4876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:02:58.832733 4876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:02:58.844539 4881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:58.844611 4882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:58.844584 4884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:02:58.845227 4876 server_base.cc:1034] running on GCE node
I20260807 07:02:58.845762 4876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:02:58.846575 4876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:02:58.847748 4876 hybrid_clock.cc:648] HybridClock initialized: now 1786086178847701 us; error 40 us; skew 500 ppm
I20260807 07:02:58.848023 4876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:02:58 dist-test-slave-ncp9 krb5kdc[4553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086178, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:02:58.851001 4876 init.cc:377] Logged in from keytab as kudu/127.21.83.2@KRBTEST.COM (short username kudu)
I20260807 07:02:58.852305 4876 webserver.cc:460] Webserver started at http://127.21.83.2:39645/ using document root <none> and password file <none>
I20260807 07:02:58.853010 4876 fs_manager.cc:359] Metadata directory not provided
I20260807 07:02:58.853129 4876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:02:58.853416 4876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:02:58.856885 4876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-1/data/instance:
uuid: "c47957eb2a2c44a287232df187b89947"
format_stamp: "Formatted at 2026-08-07 07:02:58 on dist-test-slave-ncp9"
server_key: "4b5a4619a495504488c5aae33f86080c"
server_key_iv: "d88e5e43387504d80d31f7eb43865287"
server_key_version: "encryptionkey@0"
I20260807 07:02:58.857880 4876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-1/wal/instance:
uuid: "c47957eb2a2c44a287232df187b89947"
format_stamp: "Formatted at 2026-08-07 07:02:58 on dist-test-slave-ncp9"
server_key: "4b5a4619a495504488c5aae33f86080c"
server_key_iv: "d88e5e43387504d80d31f7eb43865287"
server_key_version: "encryptionkey@0"
I20260807 07:02:58.863973 4876 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260807 07:02:58.868423 4895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:58.870150 4876 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 07:02:58.870332 4876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-1/wal
uuid: "c47957eb2a2c44a287232df187b89947"
format_stamp: "Formatted at 2026-08-07 07:02:58 on dist-test-slave-ncp9"
server_key: "4b5a4619a495504488c5aae33f86080c"
server_key_iv: "d88e5e43387504d80d31f7eb43865287"
server_key_version: "encryptionkey@0"
I20260807 07:02:58.870580 4876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:02:58.891017 4876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:02:58.894946 4876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:02:58.895182 4876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:02:58.896039 4876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:02:58.897274 4876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:02:58.897346 4876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:58.897435 4876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:02:58.897487 4876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:58.927289 4876 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.2:43679
I20260807 07:02:58.927328 5050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.2:43679 every 8 connection(s)
I20260807 07:02:58.928453 4876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
I20260807 07:02:58.931912 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 4876
I20260807 07:02:58.932113 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-1/wal/instance
I20260807 07:02:58.932672 21836 external_mini_cluster.cc:1428] Setting key 61706c338ebf7a6ea2ef80c915ac2226
Aug 07 07:02:58 dist-test-slave-ncp9 krb5kdc[4553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086178, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.2@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:02:58.947150 4872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:58.930744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:58995 (local address 127.21.83.62:35031)
0807 07:02:58.930902 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:58.930906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:58.932074 (+ 1168us) server_negotiation.cc:407] Connection header received
0807 07:02:58.934237 (+ 2163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:58.934243 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:58.934321 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:58.934404 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:58.935899 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:58.936663 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:58.938164 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:58.938578 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:58.938686 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:58.941950 (+ 3264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:58.941955 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:58.941958 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:58.941989 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:58.944345 (+ 2356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:58.945015 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:58.945020 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:58.945022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:58.945153 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:58.945549 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:58.945553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:58.945555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:58.945823 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:58.945952 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:58.946580 (+ 628us) server_negotiation.cc:299] Negotiation successful
0807 07:02:58.946823 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260807 07:02:58.948076 5054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:58.931109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:35031 (local address 127.21.83.2:58995)
0807 07:02:58.931903 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:58.931959 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:58.933989 (+ 2030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:58.934697 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:58.934705 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:58.935200 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:58.935727 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:58.935743 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:58.936810 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:58.936813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:58.938023 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:58.938036 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:58.938789 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:58.938797 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:58.939002 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:58.939608 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:58.939634 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:58.941752 (+ 2118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:58.944471 (+ 2719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:58.944479 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:58.944493 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:58.944878 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:58.945253 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:58.945258 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:58.945262 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:58.945437 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:58.945944 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:58.945951 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:58.946249 (+ 298us) client_negotiation.cc:769] Sending connection context
0807 07:02:58.946466 (+ 217us) client_negotiation.cc:240] Negotiation successful
0807 07:02:58.946935 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":247,"threads_started":1}
I20260807 07:02:58.949429 5051 heartbeater.cc:347] Connected to a master server at 127.21.83.62:35031
I20260807 07:02:58.949698 5051 heartbeater.cc:464] Registering TS with master...
I20260807 07:02:58.950373 5051 heartbeater.cc:511] Master 127.21.83.62:35031 requested a full tablet report, sending...
I20260807 07:02:58.951676 4621 ts_manager.cc:194] Registered new tserver with Master: c47957eb2a2c44a287232df187b89947 (127.21.83.2:43679)
I20260807 07:02:58.952746 4621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.2@KRBTEST.COM'} at 127.21.83.2:58995
WARNING: no policy specified for kudu/127.21.83.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab.
Entry for principal kudu/127.21.83.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab.
I20260807 07:02:58.993304 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.3:0
--local_ip_for_outbound_sockets=127.21.83.3
--webserver_interface=127.21.83.3
--webserver_port=0
--tserver_master_addrs=127.21.83.62:35031
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab
--principal=kudu/127.21.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:35061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:02:59.326876 5058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:02:59.327162 5058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:02:59.327286 5058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:02:59.334488 5058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:02:59.334589 5058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:02:59.334734 5058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.3
I20260807 07:02:59.344398 5058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:35061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab
--principal=kudu/127.21.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.83.3
--webserver_port=0
--tserver_master_addrs=127.21.83.62:35031
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.3
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:02:59.345613 5058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:02:59.346884 5058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:02:59.359246 5066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:59.359287 5064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:02:59.359474 5071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:02:59.359936 5058 server_base.cc:1034] running on GCE node
I20260807 07:02:59.360574 5058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:02:59.361564 5058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:02:59.362813 5058 hybrid_clock.cc:648] HybridClock initialized: now 1786086179362768 us; error 28 us; skew 500 ppm
I20260807 07:02:59.363184 5058 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:02:59 dist-test-slave-ncp9 krb5kdc[4553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086179, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:02:59.366488 5058 init.cc:377] Logged in from keytab as kudu/127.21.83.3@KRBTEST.COM (short username kudu)
I20260807 07:02:59.368032 5058 webserver.cc:460] Webserver started at http://127.21.83.3:40827/ using document root <none> and password file <none>
I20260807 07:02:59.368713 5058 fs_manager.cc:359] Metadata directory not provided
I20260807 07:02:59.368815 5058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:02:59.369129 5058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:02:59.372008 5058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-2/data/instance:
uuid: "3954d12ecfa840d691b9cccd294ebdd0"
format_stamp: "Formatted at 2026-08-07 07:02:59 on dist-test-slave-ncp9"
server_key: "91f61955fb2f47f472fb5423c70a4c7b"
server_key_iv: "1fb52f0ea6cc7c8ffadfdd14d5c2e8ac"
server_key_version: "encryptionkey@0"
I20260807 07:02:59.372709 5058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-2/wal/instance:
uuid: "3954d12ecfa840d691b9cccd294ebdd0"
format_stamp: "Formatted at 2026-08-07 07:02:59 on dist-test-slave-ncp9"
server_key: "91f61955fb2f47f472fb5423c70a4c7b"
server_key_iv: "1fb52f0ea6cc7c8ffadfdd14d5c2e8ac"
server_key_version: "encryptionkey@0"
I20260807 07:02:59.377715 5058 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260807 07:02:59.380860 5080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:59.382105 5058 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 07:02:59.382265 5058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-2/wal
uuid: "3954d12ecfa840d691b9cccd294ebdd0"
format_stamp: "Formatted at 2026-08-07 07:02:59 on dist-test-slave-ncp9"
server_key: "91f61955fb2f47f472fb5423c70a4c7b"
server_key_iv: "1fb52f0ea6cc7c8ffadfdd14d5c2e8ac"
server_key_version: "encryptionkey@0"
I20260807 07:02:59.382428 5058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:02:59.402438 5058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:02:59.406546 5058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:02:59.406814 5058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:02:59.407704 5058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:02:59.409008 5058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:02:59.409083 5058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:59.409205 5058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:02:59.409255 5058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:02:59.441203 5058 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.3:45687
I20260807 07:02:59.441260 5239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.3:45687 every 8 connection(s)
I20260807 07:02:59.442401 5058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
I20260807 07:02:59.443061 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 5058
I20260807 07:02:59.443251 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-2/wal/instance
I20260807 07:02:59.443778 21836 external_mini_cluster.cc:1428] Setting key bbdc337fd1056dde58d17e09ed206651
I20260807 07:02:59.463343 4867 heartbeater.cc:503] Master 127.21.83.62:35031 was elected leader, sending a full tablet report...
Aug 07 07:02:59 dist-test-slave-ncp9 krb5kdc[4553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086179, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.3@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:02:59.486198 5244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:59.450559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:33467 (local address 127.21.83.62:35031)
0807 07:02:59.464853 (+ 14294us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:59.464859 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:59.464882 (+ 23us) server_negotiation.cc:407] Connection header received
0807 07:02:59.464962 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:59.464968 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:59.465060 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:59.465203 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:59.467567 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:59.468363 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:59.470633 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:59.471307 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:59.471446 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:59.477881 (+ 6435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:59.477887 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:59.477891 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:59.477932 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:59.480960 (+ 3028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:59.481909 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:59.481916 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:59.481921 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:59.482091 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:59.482598 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:59.482605 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:59.482608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:59.482927 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:59.483082 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:59.485634 (+ 2552us) server_negotiation.cc:299] Negotiation successful
0807 07:02:59.485832 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14161,"thread_start_us":14772,"threads_started":1}
I20260807 07:02:59.486397 5242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:59.450069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:35031 (local address 127.21.83.3:33467)
0807 07:02:59.456006 (+ 5937us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:59.456062 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:59.458488 (+ 2426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:59.465598 (+ 7110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:59.465610 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:59.466410 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:02:59.467336 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:59.467369 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:59.468559 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:59.468563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:59.470434 (+ 1871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:59.470453 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:59.471608 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:59.471618 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:59.471950 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:59.474456 (+ 2506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:02:59.474491 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:02:59.477578 (+ 3087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:02:59.481165 (+ 3587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:59.481176 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:59.481192 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:59.481748 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:59.482232 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:02:59.482237 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:02:59.482241 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:02:59.482464 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:02:59.483071 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:02:59.483080 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:02:59.483477 (+ 397us) client_negotiation.cc:769] Sending connection context
0807 07:02:59.483767 (+ 290us) client_negotiation.cc:240] Negotiation successful
0807 07:02:59.484290 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5646,"thread_start_us":5442,"threads_started":1}
I20260807 07:02:59.488078 5240 heartbeater.cc:347] Connected to a master server at 127.21.83.62:35031
I20260807 07:02:59.488389 5240 heartbeater.cc:464] Registering TS with master...
I20260807 07:02:59.489328 5240 heartbeater.cc:511] Master 127.21.83.62:35031 requested a full tablet report, sending...
I20260807 07:02:59.490850 4621 ts_manager.cc:194] Registered new tserver with Master: 3954d12ecfa840d691b9cccd294ebdd0 (127.21.83.3:45687)
I20260807 07:02:59.491683 4621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.3@KRBTEST.COM'} at 127.21.83.3:33467
I20260807 07:02:59.502116 21836 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 07:02:59.519984 5244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:59.506937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59404 (local address 127.21.83.62:35031)
0807 07:02:59.507179 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:59.507187 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:59.507517 (+ 330us) server_negotiation.cc:407] Connection header received
0807 07:02:59.507663 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:59.507667 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:59.507737 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:59.507824 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:59.509393 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:59.510173 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:59.511491 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:59.511939 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:59.512055 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:59.514488 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:59.514495 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:59.514498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:59.514538 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:59.517015 (+ 2477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:59.517771 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:59.517776 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:59.517779 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:59.517932 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:59.518501 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:59.518506 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:59.518508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:59.518915 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:59.519128 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:59.519467 (+ 339us) server_negotiation.cc:299] Negotiation successful
0807 07:02:59.519680 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 07:02:59.524452 4621 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59404:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 07:02:59.531592 4621 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 07:02:59.560534 5260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:59.546021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.2:43679 (local address 127.0.0.1:50340)
0807 07:02:59.548263 (+ 2242us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:59.548319 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:59.548533 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:59.549965 (+ 1432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:59.549976 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:59.550016 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:02:59.550481 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:59.550495 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:59.554271 (+ 3776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:59.554279 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:59.556917 (+ 2638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:59.556942 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:59.559421 (+ 2479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:59.559428 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:59.559677 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:59.559742 (+ 65us) client_negotiation.cc:769] Sending connection context
0807 07:02:59.559864 (+ 122us) client_negotiation.cc:240] Negotiation successful
0807 07:02:59.560103 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1990,"thread_start_us":1549,"threads_started":1}
I20260807 07:02:59.561362 5261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:59.546537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50340 (local address 127.21.83.2:43679)
0807 07:02:59.547406 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:59.547417 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:59.548463 (+ 1046us) server_negotiation.cc:407] Connection header received
0807 07:02:59.548871 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:59.548880 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:59.549190 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:59.549405 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:02:59.551122 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:59.552960 (+ 1838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:59.557177 (+ 4217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:59.558888 (+ 1711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:59.558997 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:59.559207 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:59.560838 (+ 1631us) server_negotiation.cc:299] Negotiation successful
0807 07:02:59.561062 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":694,"thread_start_us":424,"threads_started":1}
I20260807 07:02:59.565371 4960 tablet_service.cc:1467] Processing CreateTablet for tablet 9aa048d51759423aa0a6cb5076400b07 (DEFAULT_TABLE table=test-table [id=be38f31a16fa453aacf91215c9cb5918]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:02:59.567184 4960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aa048d51759423aa0a6cb5076400b07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:02:59.568717 5264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:59.548201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38804 (local address 127.21.83.3:45687)
0807 07:02:59.558885 (+ 10684us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:59.558896 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:59.558934 (+ 38us) server_negotiation.cc:407] Connection header received
0807 07:02:59.559047 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:59.559056 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:59.559399 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:59.560553 (+ 1154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:02:59.560860 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:59.563406 (+ 2546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:59.565886 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:59.567399 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:59.567552 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:59.567944 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:59.568146 (+ 202us) server_negotiation.cc:299] Negotiation successful
0807 07:02:59.568334 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10461,"thread_start_us":11359,"threads_started":1}
I20260807 07:02:59.568723 5266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:59.547639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.3:45687 (local address 127.0.0.1:38804)
0807 07:02:59.549800 (+ 2161us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:59.549818 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:59.549929 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:59.559872 (+ 9943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:59.559877 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:59.559914 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:02:59.560324 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:59.560338 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:59.563555 (+ 3217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:59.563561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:59.565668 (+ 2107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:59.565685 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:59.567722 (+ 2037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:59.567731 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:59.567914 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:59.567952 (+ 38us) client_negotiation.cc:769] Sending connection context
0807 07:02:59.568099 (+ 147us) client_negotiation.cc:240] Negotiation successful
0807 07:02:59.568316 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2011,"thread_start_us":1787,"threads_started":1}
I20260807 07:02:59.569250 5262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:59.546533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:38597 (local address 127.0.0.1:44504)
0807 07:02:59.548868 (+ 2335us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:59.548886 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:59.549020 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:59.561418 (+ 12398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:59.561423 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:59.561452 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:02:59.561821 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:59.561836 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:59.563213 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:59.563219 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:59.565870 (+ 2651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:59.565883 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:59.568118 (+ 2235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:59.568126 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:59.568319 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:59.568351 (+ 32us) client_negotiation.cc:769] Sending connection context
0807 07:02:59.568456 (+ 105us) client_negotiation.cc:240] Negotiation successful
0807 07:02:59.568595 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2202,"thread_start_us":1438,"threads_started":1}
I20260807 07:02:59.569363 5263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:59.547329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44504 (local address 127.21.83.1:38597)
0807 07:02:59.560783 (+ 13454us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:59.560791 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:59.560820 (+ 29us) server_negotiation.cc:407] Connection header received
0807 07:02:59.560947 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:59.560954 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:59.561220 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:59.561397 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:02:59.562029 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:59.563060 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:59.566066 (+ 3006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:59.567511 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:59.567731 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:59.567953 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:59.568837 (+ 884us) server_negotiation.cc:299] Negotiation successful
0807 07:02:59.569041 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13246,"thread_start_us":14282,"threads_started":1}
I20260807 07:02:59.573119 5156 tablet_service.cc:1467] Processing CreateTablet for tablet 9aa048d51759423aa0a6cb5076400b07 (DEFAULT_TABLE table=test-table [id=be38f31a16fa453aacf91215c9cb5918]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:02:59.573112 4786 tablet_service.cc:1467] Processing CreateTablet for tablet 9aa048d51759423aa0a6cb5076400b07 (DEFAULT_TABLE table=test-table [id=be38f31a16fa453aacf91215c9cb5918]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:02:59.574793 5156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aa048d51759423aa0a6cb5076400b07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:02:59.574784 4786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aa048d51759423aa0a6cb5076400b07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:02:59.580492 5267 tablet_bootstrap.cc:492] T 9aa048d51759423aa0a6cb5076400b07 P c47957eb2a2c44a287232df187b89947: Bootstrap starting.
I20260807 07:02:59.584877 5267 tablet_bootstrap.cc:654] T 9aa048d51759423aa0a6cb5076400b07 P c47957eb2a2c44a287232df187b89947: Neither blocks nor log segments found. Creating new log.
I20260807 07:02:59.586246 5267 log.cc:824] T 9aa048d51759423aa0a6cb5076400b07 P c47957eb2a2c44a287232df187b89947: Log is configured to *not* fsync() on all Append() calls
I20260807 07:02:59.589717 5267 tablet_bootstrap.cc:492] T 9aa048d51759423aa0a6cb5076400b07 P c47957eb2a2c44a287232df187b89947: No bootstrap required, opened a new log
I20260807 07:02:59.590030 5267 ts_tablet_manager.cc:1397] T 9aa048d51759423aa0a6cb5076400b07 P c47957eb2a2c44a287232df187b89947: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260807 07:02:59.597883 5271 tablet_bootstrap.cc:492] T 9aa048d51759423aa0a6cb5076400b07 P 3954d12ecfa840d691b9cccd294ebdd0: Bootstrap starting.
I20260807 07:02:59.598080 5267 raft_consensus.cc:348] T 9aa048d51759423aa0a6cb5076400b07 P c47957eb2a2c44a287232df187b89947 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c47957eb2a2c44a287232df187b89947" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43679 } } peers { permanent_uuid: "ee846b8d4ab94ef8bf6583c398214b8b" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 38597 } } peers { permanent_uuid: "3954d12ecfa840d691b9cccd294ebdd0" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 45687 } }
I20260807 07:02:59.598897 5270 tablet_bootstrap.cc:492] T 9aa048d51759423aa0a6cb5076400b07 P ee846b8d4ab94ef8bf6583c398214b8b: Bootstrap starting.
I20260807 07:02:59.599038 5267 raft_consensus.cc:374] T 9aa048d51759423aa0a6cb5076400b07 P c47957eb2a2c44a287232df187b89947 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:02:59.599169 5267 raft_consensus.cc:729] T 9aa048d51759423aa0a6cb5076400b07 P c47957eb2a2c44a287232df187b89947 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c47957eb2a2c44a287232df187b89947, State: Initialized, Role: FOLLOWER
I20260807 07:02:59.599954 5267 consensus_queue.cc:260] T 9aa048d51759423aa0a6cb5076400b07 P c47957eb2a2c44a287232df187b89947 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c47957eb2a2c44a287232df187b89947" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43679 } } peers { permanent_uuid: "ee846b8d4ab94ef8bf6583c398214b8b" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 38597 } } peers { permanent_uuid: "3954d12ecfa840d691b9cccd294ebdd0" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 45687 } }
I20260807 07:02:59.601287 5051 heartbeater.cc:503] Master 127.21.83.62:35031 was elected leader, sending a full tablet report...
I20260807 07:02:59.603241 5270 tablet_bootstrap.cc:654] T 9aa048d51759423aa0a6cb5076400b07 P ee846b8d4ab94ef8bf6583c398214b8b: Neither blocks nor log segments found. Creating new log.
I20260807 07:02:59.604355 5271 tablet_bootstrap.cc:654] T 9aa048d51759423aa0a6cb5076400b07 P 3954d12ecfa840d691b9cccd294ebdd0: Neither blocks nor log segments found. Creating new log.
I20260807 07:02:59.604678 5270 log.cc:824] T 9aa048d51759423aa0a6cb5076400b07 P ee846b8d4ab94ef8bf6583c398214b8b: Log is configured to *not* fsync() on all Append() calls
I20260807 07:02:59.605551 5271 log.cc:824] T 9aa048d51759423aa0a6cb5076400b07 P 3954d12ecfa840d691b9cccd294ebdd0: Log is configured to *not* fsync() on all Append() calls
I20260807 07:02:59.606710 5267 ts_tablet_manager.cc:1428] T 9aa048d51759423aa0a6cb5076400b07 P c47957eb2a2c44a287232df187b89947: Time spent starting tablet: real 0.017s user 0.007s sys 0.005s
I20260807 07:02:59.608144 5270 tablet_bootstrap.cc:492] T 9aa048d51759423aa0a6cb5076400b07 P ee846b8d4ab94ef8bf6583c398214b8b: No bootstrap required, opened a new log
I20260807 07:02:59.608423 5270 ts_tablet_manager.cc:1397] T 9aa048d51759423aa0a6cb5076400b07 P ee846b8d4ab94ef8bf6583c398214b8b: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260807 07:02:59.608861 5271 tablet_bootstrap.cc:492] T 9aa048d51759423aa0a6cb5076400b07 P 3954d12ecfa840d691b9cccd294ebdd0: No bootstrap required, opened a new log
I20260807 07:02:59.609195 5271 ts_tablet_manager.cc:1397] T 9aa048d51759423aa0a6cb5076400b07 P 3954d12ecfa840d691b9cccd294ebdd0: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260807 07:02:59.615988 5270 raft_consensus.cc:348] T 9aa048d51759423aa0a6cb5076400b07 P ee846b8d4ab94ef8bf6583c398214b8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c47957eb2a2c44a287232df187b89947" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43679 } } peers { permanent_uuid: "ee846b8d4ab94ef8bf6583c398214b8b" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 38597 } } peers { permanent_uuid: "3954d12ecfa840d691b9cccd294ebdd0" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 45687 } }
I20260807 07:02:59.616541 5270 raft_consensus.cc:374] T 9aa048d51759423aa0a6cb5076400b07 P ee846b8d4ab94ef8bf6583c398214b8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:02:59.616287 5271 raft_consensus.cc:348] T 9aa048d51759423aa0a6cb5076400b07 P 3954d12ecfa840d691b9cccd294ebdd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c47957eb2a2c44a287232df187b89947" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43679 } } peers { permanent_uuid: "ee846b8d4ab94ef8bf6583c398214b8b" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 38597 } } peers { permanent_uuid: "3954d12ecfa840d691b9cccd294ebdd0" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 45687 } }
I20260807 07:02:59.616657 5270 raft_consensus.cc:729] T 9aa048d51759423aa0a6cb5076400b07 P ee846b8d4ab94ef8bf6583c398214b8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee846b8d4ab94ef8bf6583c398214b8b, State: Initialized, Role: FOLLOWER
I20260807 07:02:59.616757 5271 raft_consensus.cc:374] T 9aa048d51759423aa0a6cb5076400b07 P 3954d12ecfa840d691b9cccd294ebdd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:02:59.616940 5271 raft_consensus.cc:729] T 9aa048d51759423aa0a6cb5076400b07 P 3954d12ecfa840d691b9cccd294ebdd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3954d12ecfa840d691b9cccd294ebdd0, State: Initialized, Role: FOLLOWER
I20260807 07:02:59.617408 5270 consensus_queue.cc:260] T 9aa048d51759423aa0a6cb5076400b07 P ee846b8d4ab94ef8bf6583c398214b8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c47957eb2a2c44a287232df187b89947" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43679 } } peers { permanent_uuid: "ee846b8d4ab94ef8bf6583c398214b8b" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 38597 } } peers { permanent_uuid: "3954d12ecfa840d691b9cccd294ebdd0" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 45687 } }
I20260807 07:02:59.617573 5271 consensus_queue.cc:260] T 9aa048d51759423aa0a6cb5076400b07 P 3954d12ecfa840d691b9cccd294ebdd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c47957eb2a2c44a287232df187b89947" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43679 } } peers { permanent_uuid: "ee846b8d4ab94ef8bf6583c398214b8b" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 38597 } } peers { permanent_uuid: "3954d12ecfa840d691b9cccd294ebdd0" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 45687 } }
I20260807 07:02:59.619127 5270 ts_tablet_manager.cc:1428] T 9aa048d51759423aa0a6cb5076400b07 P ee846b8d4ab94ef8bf6583c398214b8b: Time spent starting tablet: real 0.011s user 0.004s sys 0.008s
I20260807 07:02:59.619300 5271 ts_tablet_manager.cc:1428] T 9aa048d51759423aa0a6cb5076400b07 P 3954d12ecfa840d691b9cccd294ebdd0: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260807 07:02:59.619748 5240 heartbeater.cc:503] Master 127.21.83.62:35031 was elected leader, sending a full tablet report...
W20260807 07:02:59.680318 5053 tablet.cc:2369] T 9aa048d51759423aa0a6cb5076400b07 P c47957eb2a2c44a287232df187b89947: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 07:02:59.685396 4871 tablet.cc:2369] T 9aa048d51759423aa0a6cb5076400b07 P ee846b8d4ab94ef8bf6583c398214b8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 07:02:59.706782 5241 tablet.cc:2369] T 9aa048d51759423aa0a6cb5076400b07 P 3954d12ecfa840d691b9cccd294ebdd0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 07:02:59.750993 5277 raft_consensus.cc:482] T 9aa048d51759423aa0a6cb5076400b07 P ee846b8d4ab94ef8bf6583c398214b8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:02:59.751319 5277 raft_consensus.cc:504] T 9aa048d51759423aa0a6cb5076400b07 P ee846b8d4ab94ef8bf6583c398214b8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c47957eb2a2c44a287232df187b89947" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43679 } } peers { permanent_uuid: "ee846b8d4ab94ef8bf6583c398214b8b" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 38597 } } peers { permanent_uuid: "3954d12ecfa840d691b9cccd294ebdd0" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 45687 } }
I20260807 07:02:59.753082 5277 leader_election.cc:288] T 9aa048d51759423aa0a6cb5076400b07 P ee846b8d4ab94ef8bf6583c398214b8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c47957eb2a2c44a287232df187b89947 (127.21.83.2:43679), 3954d12ecfa840d691b9cccd294ebdd0 (127.21.83.3:45687)
I20260807 07:02:59.759735 5264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:59.753543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:57123 (local address 127.21.83.3:45687)
0807 07:02:59.753761 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:59.753768 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:59.754522 (+ 754us) server_negotiation.cc:407] Connection header received
0807 07:02:59.754708 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:59.754713 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:59.754775 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:59.754851 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:02:59.755499 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:59.756219 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:59.758097 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:59.758859 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:59.758971 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:59.759082 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:59.759356 (+ 274us) server_negotiation.cc:299] Negotiation successful
0807 07:02:59.759479 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260807 07:02:59.759855 5283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:59.753606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.3:45687 (local address 127.21.83.1:57123)
0807 07:02:59.754429 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:59.754446 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:59.754573 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:59.754881 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:59.754885 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:59.754909 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:02:59.755351 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:59.755362 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:59.756349 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:59.756358 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:59.757952 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:59.757965 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:59.759064 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:59.759070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:59.759206 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:59.759233 (+ 27us) client_negotiation.cc:769] Sending connection context
0807 07:02:59.759319 (+ 86us) client_negotiation.cc:240] Negotiation successful
0807 07:02:59.759441 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"mutex_wait_us":4,"thread_start_us":441,"threads_started":1}
I20260807 07:02:59.759891 5261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:59.753615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:57395 (local address 127.21.83.2:43679)
0807 07:02:59.753822 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:59.753829 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:59.754437 (+ 608us) server_negotiation.cc:407] Connection header received
0807 07:02:59.754702 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:59.754706 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:59.754770 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:59.754844 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:02:59.755478 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:59.756171 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:59.758248 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:59.759027 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:59.759156 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:59.759249 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:59.759531 (+ 282us) server_negotiation.cc:299] Negotiation successful
0807 07:02:59.759655 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 07:02:59.760025 5282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:59.753470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.2:43679 (local address 127.21.83.1:57395)
0807 07:02:59.754313 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0807 07:02:59.754334 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 07:02:59.754494 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:02:59.754882 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:02:59.754886 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:02:59.754943 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:02:59.755336 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:59.755347 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:59.756311 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:59.756314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:59.758099 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:02:59.758111 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:02:59.759250 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:02:59.759257 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:02:59.759387 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:59.759412 (+ 25us) client_negotiation.cc:769] Sending connection context
0807 07:02:59.759492 (+ 80us) client_negotiation.cc:240] Negotiation successful
0807 07:02:59.759623 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"mutex_wait_us":69,"spinlock_wait_cycles":33664,"thread_start_us":484,"threads_started":1}
I20260807 07:02:59.760978 4985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa048d51759423aa0a6cb5076400b07" candidate_uuid: "ee846b8d4ab94ef8bf6583c398214b8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c47957eb2a2c44a287232df187b89947" is_pre_election: true
I20260807 07:02:59.760980 5181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa048d51759423aa0a6cb5076400b07" candidate_uuid: "ee846b8d4ab94ef8bf6583c398214b8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3954d12ecfa840d691b9cccd294ebdd0" is_pre_election: true
I20260807 07:02:59.761452 4985 raft_consensus.cc:2454] T 9aa048d51759423aa0a6cb5076400b07 P c47957eb2a2c44a287232df187b89947 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee846b8d4ab94ef8bf6583c398214b8b in term 0.
I20260807 07:02:59.761451 5181 raft_consensus.cc:2454] T 9aa048d51759423aa0a6cb5076400b07 P 3954d12ecfa840d691b9cccd294ebdd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee846b8d4ab94ef8bf6583c398214b8b in term 0.
I20260807 07:02:59.762066 4723 leader_election.cc:302] T 9aa048d51759423aa0a6cb5076400b07 P ee846b8d4ab94ef8bf6583c398214b8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3954d12ecfa840d691b9cccd294ebdd0, c47957eb2a2c44a287232df187b89947, ee846b8d4ab94ef8bf6583c398214b8b; no voters:
I20260807 07:02:59.762475 5277 raft_consensus.cc:2781] T 9aa048d51759423aa0a6cb5076400b07 P ee846b8d4ab94ef8bf6583c398214b8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 07:02:59.762605 5277 raft_consensus.cc:482] T 9aa048d51759423aa0a6cb5076400b07 P ee846b8d4ab94ef8bf6583c398214b8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:02:59.762720 5277 raft_consensus.cc:3037] T 9aa048d51759423aa0a6cb5076400b07 P ee846b8d4ab94ef8bf6583c398214b8b [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:02:59.764878 5277 raft_consensus.cc:504] T 9aa048d51759423aa0a6cb5076400b07 P ee846b8d4ab94ef8bf6583c398214b8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c47957eb2a2c44a287232df187b89947" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43679 } } peers { permanent_uuid: "ee846b8d4ab94ef8bf6583c398214b8b" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 38597 } } peers { permanent_uuid: "3954d12ecfa840d691b9cccd294ebdd0" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 45687 } }
I20260807 07:02:59.765507 5277 leader_election.cc:288] T 9aa048d51759423aa0a6cb5076400b07 P ee846b8d4ab94ef8bf6583c398214b8b [CANDIDATE]: Term 1 election: Requested vote from peers c47957eb2a2c44a287232df187b89947 (127.21.83.2:43679), 3954d12ecfa840d691b9cccd294ebdd0 (127.21.83.3:45687)
I20260807 07:02:59.765924 4985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa048d51759423aa0a6cb5076400b07" candidate_uuid: "ee846b8d4ab94ef8bf6583c398214b8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c47957eb2a2c44a287232df187b89947"
I20260807 07:02:59.765972 5181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa048d51759423aa0a6cb5076400b07" candidate_uuid: "ee846b8d4ab94ef8bf6583c398214b8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3954d12ecfa840d691b9cccd294ebdd0"
I20260807 07:02:59.766117 4985 raft_consensus.cc:3037] T 9aa048d51759423aa0a6cb5076400b07 P c47957eb2a2c44a287232df187b89947 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:02:59.766141 5181 raft_consensus.cc:3037] T 9aa048d51759423aa0a6cb5076400b07 P 3954d12ecfa840d691b9cccd294ebdd0 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:02:59.768244 5181 raft_consensus.cc:2454] T 9aa048d51759423aa0a6cb5076400b07 P 3954d12ecfa840d691b9cccd294ebdd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee846b8d4ab94ef8bf6583c398214b8b in term 1.
I20260807 07:02:59.768262 4985 raft_consensus.cc:2454] T 9aa048d51759423aa0a6cb5076400b07 P c47957eb2a2c44a287232df187b89947 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee846b8d4ab94ef8bf6583c398214b8b in term 1.
I20260807 07:02:59.768662 4723 leader_election.cc:302] T 9aa048d51759423aa0a6cb5076400b07 P ee846b8d4ab94ef8bf6583c398214b8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3954d12ecfa840d691b9cccd294ebdd0, c47957eb2a2c44a287232df187b89947, ee846b8d4ab94ef8bf6583c398214b8b; no voters:
I20260807 07:02:59.768977 5277 raft_consensus.cc:2781] T 9aa048d51759423aa0a6cb5076400b07 P ee846b8d4ab94ef8bf6583c398214b8b [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:02:59.769440 5277 raft_consensus.cc:686] T 9aa048d51759423aa0a6cb5076400b07 P ee846b8d4ab94ef8bf6583c398214b8b [term 1 LEADER]: Becoming Leader. State: Replica: ee846b8d4ab94ef8bf6583c398214b8b, State: Running, Role: LEADER
I20260807 07:02:59.769912 5277 consensus_queue.cc:237] T 9aa048d51759423aa0a6cb5076400b07 P ee846b8d4ab94ef8bf6583c398214b8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c47957eb2a2c44a287232df187b89947" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43679 } } peers { permanent_uuid: "ee846b8d4ab94ef8bf6583c398214b8b" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 38597 } } peers { permanent_uuid: "3954d12ecfa840d691b9cccd294ebdd0" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 45687 } }
I20260807 07:02:59.774979 4621 catalog_manager.cc:5495] T 9aa048d51759423aa0a6cb5076400b07 P ee846b8d4ab94ef8bf6583c398214b8b reported cstate change: term changed from 0 to 1, leader changed from <none> to ee846b8d4ab94ef8bf6583c398214b8b (127.21.83.1). New cstate: current_term: 1 leader_uuid: "ee846b8d4ab94ef8bf6583c398214b8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c47957eb2a2c44a287232df187b89947" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee846b8d4ab94ef8bf6583c398214b8b" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 38597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3954d12ecfa840d691b9cccd294ebdd0" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 45687 } health_report { overall_health: UNKNOWN } } }
I20260807 07:02:59.804865 5263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:59.799109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44516 (local address 127.21.83.1:38597)
0807 07:02:59.799456 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:59.799462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:59.799516 (+ 54us) server_negotiation.cc:407] Connection header received
0807 07:02:59.799822 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:59.799827 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:59.799914 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:59.800056 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:02:59.800374 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:59.801159 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:59.802589 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:59.803034 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:59.803134 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:59.803774 (+ 640us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:02:59.804350 (+ 576us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:02:59.804501 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:59.804581 (+ 80us) server_negotiation.cc:299] Negotiation successful
0807 07:02:59.804657 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 07:02:59.813319 5181 raft_consensus.cc:1261] T 9aa048d51759423aa0a6cb5076400b07 P 3954d12ecfa840d691b9cccd294ebdd0 [term 1 FOLLOWER]: Refusing update from remote peer ee846b8d4ab94ef8bf6583c398214b8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:02:59.813313 4985 raft_consensus.cc:1261] T 9aa048d51759423aa0a6cb5076400b07 P c47957eb2a2c44a287232df187b89947 [term 1 FOLLOWER]: Refusing update from remote peer ee846b8d4ab94ef8bf6583c398214b8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:02:59.814314 5287 consensus_queue.cc:1035] T 9aa048d51759423aa0a6cb5076400b07 P ee846b8d4ab94ef8bf6583c398214b8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c47957eb2a2c44a287232df187b89947" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:02:59.814640 5277 consensus_queue.cc:1035] T 9aa048d51759423aa0a6cb5076400b07 P ee846b8d4ab94ef8bf6583c398214b8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3954d12ecfa840d691b9cccd294ebdd0" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 45687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:02:59.847258 4620 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:02:59.847647 4620 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:02:59.852130 4620 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 29861f3d836b46ef9c50fbd3dee81a9f: Sending DeleteTablet for 3 replicas of tablet 9aa048d51759423aa0a6cb5076400b07
I20260807 07:02:59.853489 4960 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aa048d51759423aa0a6cb5076400b07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:02:59 UTC) from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:50340
I20260807 07:02:59.853513 4786 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aa048d51759423aa0a6cb5076400b07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:02:59 UTC) from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:44504
I20260807 07:02:59.853525 5156 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aa048d51759423aa0a6cb5076400b07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:02:59 UTC) from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:38804
I20260807 07:02:59.854853 5305 tablet_replica.cc:330] T 9aa048d51759423aa0a6cb5076400b07 P 3954d12ecfa840d691b9cccd294ebdd0: stopping tablet replica
I20260807 07:02:59.855365 5305 raft_consensus.cc:2229] T 9aa048d51759423aa0a6cb5076400b07 P 3954d12ecfa840d691b9cccd294ebdd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:02:59.855588 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 4698
I20260807 07:02:59.855970 5305 raft_consensus.cc:2258] T 9aa048d51759423aa0a6cb5076400b07 P 3954d12ecfa840d691b9cccd294ebdd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:02:59.858172 5305 ts_tablet_manager.cc:1905] T 9aa048d51759423aa0a6cb5076400b07 P 3954d12ecfa840d691b9cccd294ebdd0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:02:59.866374 5305 ts_tablet_manager.cc:1918] T 9aa048d51759423aa0a6cb5076400b07 P 3954d12ecfa840d691b9cccd294ebdd0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:02:59.866731 5305 log.cc:1196] T 9aa048d51759423aa0a6cb5076400b07 P 3954d12ecfa840d691b9cccd294ebdd0: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-2/wal/wals/9aa048d51759423aa0a6cb5076400b07
I20260807 07:02:59.867439 5305 ts_tablet_manager.cc:1939] T 9aa048d51759423aa0a6cb5076400b07 P 3954d12ecfa840d691b9cccd294ebdd0: Deleting consensus metadata
I20260807 07:02:59.869609 5303 tablet_replica.cc:330] T 9aa048d51759423aa0a6cb5076400b07 P c47957eb2a2c44a287232df187b89947: stopping tablet replica
I20260807 07:02:59.870106 5303 raft_consensus.cc:2229] T 9aa048d51759423aa0a6cb5076400b07 P c47957eb2a2c44a287232df187b89947 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:02:59.870647 5303 raft_consensus.cc:2258] T 9aa048d51759423aa0a6cb5076400b07 P c47957eb2a2c44a287232df187b89947 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:02:59.871727 4600 catalog_manager.cc:4841] TS 3954d12ecfa840d691b9cccd294ebdd0 (127.21.83.3:45687): tablet 9aa048d51759423aa0a6cb5076400b07 (table test-table [id=be38f31a16fa453aacf91215c9cb5918]) successfully deleted
I20260807 07:02:59.873082 5304 tablet_replica.cc:330] T 9aa048d51759423aa0a6cb5076400b07 P ee846b8d4ab94ef8bf6583c398214b8b: stopping tablet replica
I20260807 07:02:59.873550 5304 raft_consensus.cc:2229] T 9aa048d51759423aa0a6cb5076400b07 P ee846b8d4ab94ef8bf6583c398214b8b [term 1 LEADER]: Raft consensus shutting down.
I20260807 07:02:59.874500 5304 raft_consensus.cc:2258] T 9aa048d51759423aa0a6cb5076400b07 P ee846b8d4ab94ef8bf6583c398214b8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:02:59.874872 5303 ts_tablet_manager.cc:1905] T 9aa048d51759423aa0a6cb5076400b07 P c47957eb2a2c44a287232df187b89947: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:02:59.882896 5303 ts_tablet_manager.cc:1918] T 9aa048d51759423aa0a6cb5076400b07 P c47957eb2a2c44a287232df187b89947: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Aug 07 07:02:59 dist-test-slave-ncp9 krb5kdc[4553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.1@KRBTEST.COM
I20260807 07:02:59.883266 5303 log.cc:1196] T 9aa048d51759423aa0a6cb5076400b07 P c47957eb2a2c44a287232df187b89947: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-1/wal/wals/9aa048d51759423aa0a6cb5076400b07
I20260807 07:02:59.883252 5304 ts_tablet_manager.cc:1905] T 9aa048d51759423aa0a6cb5076400b07 P ee846b8d4ab94ef8bf6583c398214b8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:02:59.883949 5303 ts_tablet_manager.cc:1939] T 9aa048d51759423aa0a6cb5076400b07 P c47957eb2a2c44a287232df187b89947: Deleting consensus metadata
I20260807 07:02:59.886349 4601 catalog_manager.cc:4841] TS c47957eb2a2c44a287232df187b89947 (127.21.83.2:43679): tablet 9aa048d51759423aa0a6cb5076400b07 (table test-table [id=be38f31a16fa453aacf91215c9cb5918]) successfully deleted
I20260807 07:02:59.891714 5304 ts_tablet_manager.cc:1918] T 9aa048d51759423aa0a6cb5076400b07 P ee846b8d4ab94ef8bf6583c398214b8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:02:59.891932 5304 log.cc:1196] T 9aa048d51759423aa0a6cb5076400b07 P ee846b8d4ab94ef8bf6583c398214b8b: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786086078080656-21836-0/minicluster-data/ts-0/wal/wals/9aa048d51759423aa0a6cb5076400b07
I20260807 07:02:59.892551 5304 ts_tablet_manager.cc:1939] T 9aa048d51759423aa0a6cb5076400b07 P ee846b8d4ab94ef8bf6583c398214b8b: Deleting consensus metadata
I20260807 07:02:59.893625 5263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:02:59.858686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44520 (local address 127.21.83.1:38597)
0807 07:02:59.858894 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0807 07:02:59.858900 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:02:59.868016 (+ 9116us) server_negotiation.cc:407] Connection header received
0807 07:02:59.868352 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:02:59.868359 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:02:59.868473 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:02:59.868551 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:02:59.871102 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:59.872403 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:59.877210 (+ 4807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:02:59.877925 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:02:59.878033 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:02:59.884428 (+ 6395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:02:59.884446 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:02:59.884469 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:02:59.884530 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:02:59.889490 (+ 4960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:59.890832 (+ 1342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:59.890843 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:59.890847 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:59.891060 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:02:59.891592 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:02:59.891598 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:02:59.891601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:02:59.892156 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:02:59.892511 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0807 07:02:59.892957 (+ 446us) server_negotiation.cc:299] Negotiation successful
0807 07:02:59.893253 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 07:02:59.894491 4861 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:02:59.894905 4598 catalog_manager.cc:4841] TS ee846b8d4ab94ef8bf6583c398214b8b (127.21.83.1:38597): tablet 9aa048d51759423aa0a6cb5076400b07 (table test-table [id=be38f31a16fa453aacf91215c9cb5918]) successfully deleted
I20260807 07:02:59.996922 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 4698
I20260807 07:03:00.023651 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 4876
Aug 07 07:03:00 dist-test-slave-ncp9 krb5kdc[4553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.2@KRBTEST.COM
I20260807 07:03:00.040556 5261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:03:00.024641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50354 (local address 127.21.83.2:43679)
0807 07:03:00.024842 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0807 07:03:00.024847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:03:00.024875 (+ 28us) server_negotiation.cc:407] Connection header received
0807 07:03:00.025061 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:03:00.025065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:03:00.025159 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:03:00.025278 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:03:00.026777 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:03:00.027593 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:03:00.029137 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:03:00.029571 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:03:00.029700 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:03:00.034103 (+ 4403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:03:00.034123 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:03:00.034140 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:03:00.034194 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:03:00.037174 (+ 2980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:03:00.038127 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:03:00.038139 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:03:00.038142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:03:00.038500 (+ 358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:03:00.038999 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:03:00.039002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:03:00.039005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:03:00.039498 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:03:00.039758 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0807 07:03:00.040043 (+ 285us) server_negotiation.cc:299] Negotiation successful
0807 07:03:00.040249 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 07:03:00.041378 5046 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:03:00.134929 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 4876
I20260807 07:03:00.160015 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 5058
Aug 07 07:03:00 dist-test-slave-ncp9 krb5kdc[4553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.3@KRBTEST.COM
I20260807 07:03:00.176270 5264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:03:00.160887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38818 (local address 127.21.83.3:45687)
0807 07:03:00.161081 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0807 07:03:00.161086 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:03:00.161107 (+ 21us) server_negotiation.cc:407] Connection header received
0807 07:03:00.161235 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:03:00.161240 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:03:00.161310 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:03:00.161396 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:03:00.162760 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:03:00.163555 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:03:00.164884 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:03:00.165463 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:03:00.165587 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:03:00.170142 (+ 4555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:03:00.170162 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:03:00.170180 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:03:00.170226 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:03:00.173233 (+ 3007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:03:00.174187 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:03:00.174196 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:03:00.174198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:03:00.174350 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:03:00.174704 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:03:00.174707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:03:00.174709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:03:00.175126 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:03:00.175367 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0807 07:03:00.175714 (+ 347us) server_negotiation.cc:299] Negotiation successful
0807 07:03:00.175906 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260807 07:03:00.177122 5234 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:03:00.268793 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 5058
I20260807 07:03:00.293395 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 4573
I20260807 07:03:00.294605 4658 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:03:00.408157 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 4573
2026-08-07T07:03:00Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4448 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 07:03:00 dist-test-slave-ncp9 krb5kdc[5317](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 07:03:00 dist-test-slave-ncp9 krb5kdc[5317](info): set up 2 sockets
Aug 07 07:03:00 dist-test-slave-ncp9 krb5kdc[5317](info): commencing operation
krb5kdc: starting...
W20260807 07:03:01.644367 21836 mini_kdc.cc:121] Time spent starting KDC: real 1.184s user 0.000s sys 0.052s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Aug 07 07:03:01 dist-test-slave-ncp9 krb5kdc[5317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T07:03:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:03:01Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.21.83.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.21.83.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.21.83.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/krb5kdc/rangeradmin_127.21.83.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.21.83.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/krb5kdc/rangeradmin_127.21.83.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.21.83.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.21.83.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.21.83.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/krb5kdc/rangerlookup_127.21.83.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.21.83.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/krb5kdc/rangerlookup_127.21.83.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.83.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.21.83.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.83.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/krb5kdc/HTTP_127.21.83.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.83.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/krb5kdc/HTTP_127.21.83.20@KRBTEST.COM.keytab.
I20260807 07:03:02.077579 21836 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskyUa4Vb/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/postgres -l logfile start
2026-08-07 07:03:04.333 UTC [5359] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-07 07:03:04.333 UTC [5359] LOG: listening on IPv4 address "127.21.83.20", port 44489
2026-08-07 07:03:04.336 UTC [5359] LOG: listening on Unix socket "/tmp/.s.PGSQL.44489"
2026-08-07 07:03:04.359 UTC [5363] LOG: database system was shut down at 2026-08-07 07:03:03 UTC
2026-08-07 07:03:04.362 UTC [5359] LOG: database system is ready to accept connections
I20260807 07:03:05.429481 21836 mini_postgres.cc:96] Postgres bound to 44489
2026-08-07 07:03:05.457 UTC [5376] FATAL: database "slave" does not exist
127.21.83.20:44489 - accepting connections
I20260807 07:03:05.458704 21836 mini_ranger.cc:161] Starting Ranger...
I20260807 07:03:05.495910 21836 mini_ranger.cc:84] Created miniranger Postgres user
I20260807 07:03:05.916833 21836 mini_ranger.cc:87] Created ranger Postgres database
I20260807 07:03:05.917013 21836 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ranger-admin
2026-08-07 07:03:06,276 [I] DB FLAVOR :POSTGRES
2026-08-07 07:03:06,276 [I] --------- Verifying Ranger DB connection ---------
2026-08-07 07:03:06,277 [I] Checking connection..
2026-08-07 07:03:06,277 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyUa4Vb/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyUa4Vb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.83.20:44489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-07 07:03:06,542 [I] Checking connection passed.
2026-08-07 07:03:06,542 [I] --------- Verifying version history table ---------
2026-08-07 07:03:06,542 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyUa4Vb/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyUa4Vb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.83.20:44489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 07:03:06,827 [I] Table x_db_version_h does not exist in database ranger
2026-08-07 07:03:06,827 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyUa4Vb/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyUa4Vb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.83.20:44489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 07:03:07,091 [I] Table x_db_version_h does not exist in database ranger
2026-08-07 07:03:07,091 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-07 07:03:07,091 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyUa4Vb/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyUa4Vb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.83.20:44489/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyUa4Vb/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-07 07:03:07.347 UTC [5474] WARNING: there is no transaction in progress
2026-08-07 07:03:07,356 [I] create_dbversion_catalog.sql file imported successfully
2026-08-07 07:03:07,356 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyUa4Vb/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyUa4Vb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.83.20:44489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 07:03:07,600 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-07 07:03:07,600 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-07 07:03:07,600 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyUa4Vb/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyUa4Vb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.83.20:44489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-07 07:03:07,852 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyUa4Vb/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyUa4Vb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.83.20:44489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-07 07:03:08,100 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyUa4Vb/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyUa4Vb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.83.20:44489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-ncp9.c.gcp-upstream.internal','N') ;"
2026-08-07 07:03:08,347 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyUa4Vb/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyUa4Vb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.83.20:44489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-07 07:03:08,587 [I] Table x_portal_user does not exist in database ranger
2026-08-07 07:03:08,587 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyUa4Vb/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyUa4Vb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.83.20:44489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-07 07:03:08,831 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-07 07:03:08,831 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyUa4Vb/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyUa4Vb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.83.20:44489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-07 07:03:09,076 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyUa4Vb/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyUa4Vb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.83.20:44489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-07 07:03:09,321 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-07 07:03:09,321 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyUa4Vb/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyUa4Vb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.83.20:44489/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyUa4Vb/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-07 07:03:10.253 UTC [5634] WARNING: there is no transaction in progress
2026-08-07 07:03:10.268 UTC [5634] WARNING: there is no transaction in progress
2026-08-07 07:03:10.284 UTC [5634] WARNING: there is no transaction in progress
2026-08-07 07:03:10.299 UTC [5634] WARNING: there is no transaction in progress
2026-08-07 07:03:10.313 UTC [5634] WARNING: there is no transaction in progress
2026-08-07 07:03:10.328 UTC [5634] WARNING: there is no transaction in progress
2026-08-07 07:03:10.379 UTC [5634] WARNING: there is no transaction in progress
2026-08-07 07:03:10.386 UTC [5634] WARNING: there is no transaction in progress
2026-08-07 07:03:10.400 UTC [5634] WARNING: there is no transaction in progress
2026-08-07 07:03:10.410 UTC [5634] WARNING: there is no transaction in progress
2026-08-07 07:03:10.420 UTC [5634] WARNING: there is no transaction in progress
2026-08-07 07:03:10.434 UTC [5634] WARNING: there is no transaction in progress
2026-08-07 07:03:10.443 UTC [5634] WARNING: there is no transaction in progress
2026-08-07 07:03:10.449 UTC [5634] WARNING: there is no transaction in progress
2026-08-07 07:03:10.454 UTC [5634] WARNING: there is no transaction in progress
2026-08-07 07:03:10.460 UTC [5634] WARNING: there is no transaction in progress
2026-08-07 07:03:11,020 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-07 07:03:11,020 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyUa4Vb/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyUa4Vb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.83.20:44489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-07 07:03:11,259 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-07 07:03:11,259 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyUa4Vb/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyUa4Vb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.83.20:44489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-07 07:03:11,510 [I] Table x_portal_user already exists in database 'ranger'
2026-08-07 07:03:11,510 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyUa4Vb/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyUa4Vb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.83.20:44489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-07 07:03:11,752 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-07 07:03:11,753 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyUa4Vb/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyUa4Vb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.83.20:44489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-07 07:03:11,988 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyUa4Vb/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyUa4Vb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.83.20:44489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-07 07:03:12,219 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyUa4Vb/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyUa4Vb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.83.20:44489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-ncp9.c.gcp-upstream.internal';"
2026-08-07 07:03:12,448 [I] CORE_DB_SCHEMA import status has been updated
2026-08-07 07:03:12,449 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyUa4Vb/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyUa4Vb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.83.20:44489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-07 07:03:12,691 [I] DB_PATCHES have already been applied
I20260807 07:03:12.697261 21836 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ranger-admin:/tmp/dist-test-taskyUa4Vb/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskyUa4Vb/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskyUa4Vb/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyUa4Vb/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260807 07:03:12.697402 21836 mini_ranger.cc:193] Using host: 127.21.83.20
I20260807 07:03:12.721078 21836 mini_ranger.cc:227] Ranger admin URL: http://127.21.83.20:55479
Aug 07, 2026 7:03:13 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 07, 2026 7:03:13 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 07, 2026 7:03:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ranger-admin/ews/webapp
Aug 07, 2026 7:03:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 07, 2026 7:03:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 07, 2026 7:03:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ranger-admin/ews/webapp].
Aug 07, 2026 7:03:13 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260807 07:03:13.941033 21836 mini_ranger.cc:160] Time spent starting Ranger: real 8.482s user 0.005s sys 0.070s
Aug 07, 2026 7:03:33 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 07, 2026 7:03:34 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 07, 2026 7:03:34 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 07, 2026 7:03:34 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 07, 2026 7:03:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 07, 2026 7:03:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 07, 2026 7:03:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 07, 2026 7:03:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 07, 2026 7:03:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 07, 2026 7:03:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 07, 2026 7:03:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 07, 2026 7:03:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 07, 2026 7:03:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 07, 2026 7:03:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 07, 2026 7:03:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 07, 2026 7:03:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 07, 2026 7:03:34 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 07, 2026 7:03:35 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260807 07:03:35.621085 21836 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.21.83.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.21.83.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.21.83.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/krb5kdc/rangerkms_127.21.83.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.21.83.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/krb5kdc/rangerkms_127.21.83.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.83.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.21.83.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.83.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/krb5kdc/HTTP_127.21.83.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.83.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/krb5kdc/HTTP_127.21.83.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 07 07:03:35 dist-test-slave-ncp9 krb5kdc[5317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086215, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260807 07:03:35.834808 21836 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260807 07:03:35.861555 21836 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260807 07:03:36.594907 21836 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260807 07:03:36.595042 21836 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ranger-kms
I20260807 07:03:36.595187 21836 mini_ranger_kms.cc:227] Using postgres at 127.21.83.20:44489
2026-08-07 07:03:37,137 [I] DB FLAVOR :POSTGRES
2026-08-07 07:03:37,137 [I] --------- Verifying Ranger DB connection ---------
2026-08-07 07:03:37,138 [I] Checking connection
2026-08-07 07:03:37,138 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyUa4Vb/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.83.20:44489/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-07 07:03:37,369 [I] connection success
2026-08-07 07:03:37,370 [I] --------- Verifying Ranger DB tables ---------
2026-08-07 07:03:37,370 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-07 07:03:37,370 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyUa4Vb/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.83.20:44489/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-07 07:03:37,614 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-07 07:03:37,614 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-07 07:03:37,615 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-07 07:03:37,615 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyUa4Vb/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.83.20:44489/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-07 07:03:37,892 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260807 07:03:38.348039 21836 mini_ranger_kms.cc:310] Created kms service
I20260807 07:03:38.544101 21836 mini_ranger_kms.cc:326] Created kudu user
I20260807 07:03:38.593448 21836 mini_ranger_kms.cc:343] Created rangerkms user
Aug 07 07:03:38 dist-test-slave-ncp9 krb5kdc[5317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086215, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.21.83.20@KRBTEST.COM
I20260807 07:03:38.981499 21836 mini_ranger_kms.cc:384] Added ranger policy
I20260807 07:03:38.982700 21836 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ranger-kms:/tmp/dist-test-taskyUa4Vb/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyUa4Vb/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskyUa4Vb/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260807 07:03:38.982771 21836 mini_ranger_kms.cc:240] Using host: 127.21.83.20
I20260807 07:03:39.018487 21836 mini_ranger_kms.cc:276] Ranger KMS PID: 5901
I20260807 07:03:39.018671 21836 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.21.83.20:34963
Aug 07, 2026 7:03:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 07, 2026 7:03:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 07, 2026 7:03:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 07, 2026 7:03:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 07, 2026 7:03:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ranger-kms/ews/webapp].
Aug 07, 2026 7:03:40 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 07 07:03:41 dist-test-slave-ncp9 krb5kdc[5317](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786086221, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.83.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:03:41.549602 21836 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.715s user 0.123s sys 0.478s
I20260807 07:03:41.550027 21836 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260807 07:03:41.550276 21836 mini_ranger_kms.cc:401] 127.21.83.20:34963/kms/v1/keys
Aug 07 07:03:41 dist-test-slave-ncp9 krb5kdc[5317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086221, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.83.20@KRBTEST.COM for HTTP/127.21.83.20@KRBTEST.COM
Aug 07 07:03:42 dist-test-slave-ncp9 krb5kdc[5317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086221, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.83.20@KRBTEST.COM for HTTP/127.21.83.20@KRBTEST.COM
Aug 07 07:03:44 dist-test-slave-ncp9 krb5kdc[5317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.21.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
I20260807 07:03:44.452740 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:40861
--webserver_interface=127.21.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:34227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:40861
--ranger_config_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.83.20:34963/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:03:45.036931 5982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:03:45.037261 5982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:03:45.037390 5982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:03:45.046588 5982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:03:45.046725 5982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:03:45.046828 5982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:03:45.046944 5982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:03:45.047057 5982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:03:45.059358 5982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:34227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/master-0/wal
--ranger_kms_url=127.21.83.20:34963/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:40861
--ranger_config_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:40861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:03:45.060757 5982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:03:45.062254 5982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:03:45.077522 5988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:03:45.077680 5990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:03:45.077700 5992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:03:45.079015 5982 server_base.cc:1034] running on GCE node
I20260807 07:03:45.079768 5982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:03:45.081112 5982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:03:45.082376 5982 hybrid_clock.cc:648] HybridClock initialized: now 1786086225082272 us; error 32 us; skew 500 ppm
I20260807 07:03:45.082855 5982 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:03:45 dist-test-slave-ncp9 krb5kdc[5317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086225, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:03:45.087224 5982 init.cc:377] Logged in from keytab as kudu/127.21.83.62@KRBTEST.COM (short username kudu)
I20260807 07:03:45.089337 5982 webserver.cc:460] Webserver started at http://127.21.83.62:44953/ using document root <none> and password file <none>
I20260807 07:03:45.090188 5982 fs_manager.cc:359] Metadata directory not provided
I20260807 07:03:45.090375 5982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:03:45.090776 5982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 07:03:45 dist-test-slave-ncp9 krb5kdc[5317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086225, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for HTTP/127.21.83.20@KRBTEST.COM
I20260807 07:03:45.155418 5982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "179f918925194d11b0c9c8505bac2789"
format_stamp: "Formatted at 2026-08-07 07:03:45 on dist-test-slave-ncp9"
server_key: "4e28a81e8130802a1067936fbf250926"
server_key_iv: "b573ffa7876ef2c234ca876a641b77ba"
server_key_version: "kuduclusterkey@0"
I20260807 07:03:45.156735 5982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "179f918925194d11b0c9c8505bac2789"
format_stamp: "Formatted at 2026-08-07 07:03:45 on dist-test-slave-ncp9"
server_key: "4e28a81e8130802a1067936fbf250926"
server_key_iv: "b573ffa7876ef2c234ca876a641b77ba"
server_key_version: "kuduclusterkey@0"
I20260807 07:03:45.165548 5982 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.002s
I20260807 07:03:45.186739 6003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:03:45.188271 5982 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20260807 07:03:45.188583 5982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "179f918925194d11b0c9c8505bac2789"
format_stamp: "Formatted at 2026-08-07 07:03:45 on dist-test-slave-ncp9"
server_key: "4e28a81e8130802a1067936fbf250926"
server_key_iv: "b573ffa7876ef2c234ca876a641b77ba"
server_key_version: "kuduclusterkey@0"
I20260807 07:03:45.188920 5982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:03:45.205435 5982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:03:45.209949 5982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:03:45.210364 5982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:03:45.247926 5982 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.62:40861
I20260807 07:03:45.248020 6082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.62:40861 every 8 connection(s)
I20260807 07:03:45.249579 5982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:03:45.254232 6083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:03:45.256222 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 5982
I20260807 07:03:45.256434 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:03:45.257210 21836 external_mini_cluster.cc:1428] Setting key 64028234ab1aaa003a4db945950f230c
I20260807 07:03:45.272282 6083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 179f918925194d11b0c9c8505bac2789: Bootstrap starting.
Aug 07 07:03:45 dist-test-slave-ncp9 krb5kdc[5317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:03:45.277400 6083 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 179f918925194d11b0c9c8505bac2789: Neither blocks nor log segments found. Creating new log.
I20260807 07:03:45.279433 6083 log.cc:824] T 00000000000000000000000000000000 P 179f918925194d11b0c9c8505bac2789: Log is configured to *not* fsync() on all Append() calls
I20260807 07:03:45.284049 6083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 179f918925194d11b0c9c8505bac2789: No bootstrap required, opened a new log
I20260807 07:03:45.285135 6090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:03:45.259281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51694 (local address 127.21.83.62:40861)
0807 07:03:45.260630 (+ 1349us) server_negotiation.cc:206] Beginning negotiation
0807 07:03:45.260641 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 07:03:45.260709 (+ 68us) server_negotiation.cc:407] Connection header received
0807 07:03:45.263127 (+ 2418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:03:45.263149 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:03:45.263543 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:03:45.263841 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:03:45.265624 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:03:45.267495 (+ 1871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:03:45.269357 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:03:45.270126 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:03:45.270266 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:03:45.275368 (+ 5102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:03:45.275376 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:03:45.275390 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:03:45.275430 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:03:45.279898 (+ 4468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:03:45.280929 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:03:45.280939 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:03:45.280947 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:03:45.281219 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:03:45.281659 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:03:45.281664 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:03:45.281666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:03:45.282481 (+ 815us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:03:45.282857 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0807 07:03:45.283281 (+ 424us) server_negotiation.cc:299] Negotiation successful
0807 07:03:45.283705 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1067,"thread_start_us":777,"threads_started":1}
I20260807 07:03:45.295971 6083 raft_consensus.cc:348] T 00000000000000000000000000000000 P 179f918925194d11b0c9c8505bac2789 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179f918925194d11b0c9c8505bac2789" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 40861 } }
I20260807 07:03:45.296450 6083 raft_consensus.cc:374] T 00000000000000000000000000000000 P 179f918925194d11b0c9c8505bac2789 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:03:45.296594 6083 raft_consensus.cc:729] T 00000000000000000000000000000000 P 179f918925194d11b0c9c8505bac2789 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 179f918925194d11b0c9c8505bac2789, State: Initialized, Role: FOLLOWER
I20260807 07:03:45.297338 6083 consensus_queue.cc:260] T 00000000000000000000000000000000 P 179f918925194d11b0c9c8505bac2789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179f918925194d11b0c9c8505bac2789" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 40861 } }
I20260807 07:03:45.297627 6083 raft_consensus.cc:388] T 00000000000000000000000000000000 P 179f918925194d11b0c9c8505bac2789 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:03:45.297808 6083 raft_consensus.cc:482] T 00000000000000000000000000000000 P 179f918925194d11b0c9c8505bac2789 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:03:45.298087 6083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 179f918925194d11b0c9c8505bac2789 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:03:45.301113 6083 raft_consensus.cc:504] T 00000000000000000000000000000000 P 179f918925194d11b0c9c8505bac2789 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179f918925194d11b0c9c8505bac2789" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 40861 } }
I20260807 07:03:45.301795 6083 leader_election.cc:302] T 00000000000000000000000000000000 P 179f918925194d11b0c9c8505bac2789 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 179f918925194d11b0c9c8505bac2789; no voters:
I20260807 07:03:45.302903 6083 leader_election.cc:288] T 00000000000000000000000000000000 P 179f918925194d11b0c9c8505bac2789 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 07:03:45.303109 6094 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 179f918925194d11b0c9c8505bac2789 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:03:45.304831 6094 raft_consensus.cc:686] T 00000000000000000000000000000000 P 179f918925194d11b0c9c8505bac2789 [term 1 LEADER]: Becoming Leader. State: Replica: 179f918925194d11b0c9c8505bac2789, State: Running, Role: LEADER
I20260807 07:03:45.305539 6094 consensus_queue.cc:237] T 00000000000000000000000000000000 P 179f918925194d11b0c9c8505bac2789 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179f918925194d11b0c9c8505bac2789" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 40861 } }
I20260807 07:03:45.306026 6083 sys_catalog.cc:564] T 00000000000000000000000000000000 P 179f918925194d11b0c9c8505bac2789 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:03:45.309199 6096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 179f918925194d11b0c9c8505bac2789 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 179f918925194d11b0c9c8505bac2789. Latest consensus state: current_term: 1 leader_uuid: "179f918925194d11b0c9c8505bac2789" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179f918925194d11b0c9c8505bac2789" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 40861 } } }
I20260807 07:03:45.309810 6096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 179f918925194d11b0c9c8505bac2789 [sys.catalog]: This master's current role is: LEADER
I20260807 07:03:45.309585 6095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 179f918925194d11b0c9c8505bac2789 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "179f918925194d11b0c9c8505bac2789" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179f918925194d11b0c9c8505bac2789" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 40861 } } }
I20260807 07:03:45.310137 6095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 179f918925194d11b0c9c8505bac2789 [sys.catalog]: This master's current role is: LEADER
I20260807 07:03:45.312189 6100 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:03:45.320683 6100 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:03:45.321758 6083 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260807 07:03:45.349192 6100 catalog_manager.cc:1347] Generated new cluster ID: c2436cd0e02c408191225c7787f9ca09
I20260807 07:03:45.349345 6100 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:03:45.366469 6100 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:03:45.368536 6100 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:03:45.387916 6100 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 179f918925194d11b0c9c8505bac2789: Generated new TSK 0
I20260807 07:03:45.389216 6100 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.83.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab.
Entry for principal kudu/127.21.83.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab.
I20260807 07:03:46.298791 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.83.20:34963/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.1:0
--local_ip_for_outbound_sockets=127.21.83.1
--webserver_interface=127.21.83.1
--webserver_port=0
--tserver_master_addrs=127.21.83.62:40861
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab
--principal=kudu/127.21.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:34227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 07:03:46 dist-test-slave-ncp9 krb5kdc[5317](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786086226, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260807 07:03:47.054178 6154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:03:47.054504 6154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:03:47.054718 6154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:03:47.068921 6154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:03:47.069238 6154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:03:47.069604 6154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.1
I20260807 07:03:47.086514 6154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:34227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.21.83.20:34963/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab
--principal=kudu/127.21.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.83.1
--webserver_port=0
--tserver_master_addrs=127.21.83.62:40861
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.1
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:03:47.088217 6154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:03:47.089848 6154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:03:47.118408 6171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:03:47.121940 6170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:03:47.127733 6154 server_base.cc:1034] running on GCE node
W20260807 07:03:47.128381 6173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:03:47.129240 6154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:03:47.132494 6154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:03:47.133841 6154 hybrid_clock.cc:648] HybridClock initialized: now 1786086227133667 us; error 93 us; skew 500 ppm
I20260807 07:03:47.134330 6154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:03:47 dist-test-slave-ncp9 krb5kdc[5317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086227, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:03:47.146024 6154 init.cc:377] Logged in from keytab as kudu/127.21.83.1@KRBTEST.COM (short username kudu)
I20260807 07:03:47.154513 6154 webserver.cc:460] Webserver started at http://127.21.83.1:40071/ using document root <none> and password file <none>
I20260807 07:03:47.155552 6154 fs_manager.cc:359] Metadata directory not provided
I20260807 07:03:47.155809 6154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:03:47.156306 6154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 07:03:47 dist-test-slave-ncp9 krb5kdc[5317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086227, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.1@KRBTEST.COM for HTTP/127.21.83.20@KRBTEST.COM
I20260807 07:03:47.185597 6154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-0/data/instance:
uuid: "bbf7de1958f343069d4339a56a592be5"
format_stamp: "Formatted at 2026-08-07 07:03:47 on dist-test-slave-ncp9"
server_key: "dd80e566a80e7cb26a01b9d6cd539e89"
server_key_iv: "1c983b2b7af52f48b23503870f92caf6"
server_key_version: "kuduclusterkey@0"
I20260807 07:03:47.187291 6154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-0/wal/instance:
uuid: "bbf7de1958f343069d4339a56a592be5"
format_stamp: "Formatted at 2026-08-07 07:03:47 on dist-test-slave-ncp9"
server_key: "dd80e566a80e7cb26a01b9d6cd539e89"
server_key_iv: "1c983b2b7af52f48b23503870f92caf6"
server_key_version: "kuduclusterkey@0"
I20260807 07:03:47.197086 6154 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.004s sys 0.004s
I20260807 07:03:47.251294 6182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:03:47.253859 6154 fs_manager.cc:725] Time spent opening block manager: real 0.024s user 0.004s sys 0.012s
Aug 07 07:03:47 dist-test-slave-ncp9 krb5kdc[5317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086226, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for HTTP/127.21.83.20@KRBTEST.COM
I20260807 07:03:47.254287 6154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-0/wal
uuid: "bbf7de1958f343069d4339a56a592be5"
format_stamp: "Formatted at 2026-08-07 07:03:47 on dist-test-slave-ncp9"
server_key: "dd80e566a80e7cb26a01b9d6cd539e89"
server_key_iv: "1c983b2b7af52f48b23503870f92caf6"
server_key_version: "kuduclusterkey@0"
I20260807 07:03:47.254716 6154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:03:47.367065 6154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:03:47.374979 6154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:03:47.375686 6154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:03:47.378237 6154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:03:47.381538 6154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:03:47.381728 6154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:03:47.382030 6154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:03:47.382227 6154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:03:47.587363 6154 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.1:42823
I20260807 07:03:47.587569 6351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.1:42823 every 8 connection(s)
I20260807 07:03:47.590020 6154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
I20260807 07:03:47.599081 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 6154
I20260807 07:03:47.599350 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-0/wal/instance
I20260807 07:03:47.600505 21836 external_mini_cluster.cc:1428] Setting key f7aacf4c82245698402b93fce779b4a3
Aug 07 07:03:47 dist-test-slave-ncp9 krb5kdc[5317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086227, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.1@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:03:47.632021 6359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:03:47.596813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40861 (local address 127.21.83.1:46927)
0807 07:03:47.599450 (+ 2637us) negotiation.cc:107] Waiting for socket to connect
0807 07:03:47.599505 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0807 07:03:47.602140 (+ 2635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:03:47.603133 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:03:47.603148 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:03:47.604064 (+ 916us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:03:47.605270 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:03:47.605295 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:03:47.606862 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:03:47.606866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:03:47.609851 (+ 2985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:03:47.609870 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:03:47.611784 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:03:47.611800 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:03:47.612307 (+ 507us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:03:47.613809 (+ 1502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:03:47.613841 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:03:47.618740 (+ 4899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:03:47.625465 (+ 6725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:03:47.625477 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:03:47.625513 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:03:47.626383 (+ 870us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:03:47.627035 (+ 652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:03:47.627040 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:03:47.627044 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:03:47.627507 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:03:47.628441 (+ 934us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:03:47.628451 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:03:47.628942 (+ 491us) client_negotiation.cc:769] Sending connection context
0807 07:03:47.629320 (+ 378us) client_negotiation.cc:240] Negotiation successful
0807 07:03:47.630092 (+ 772us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2179,"thread_start_us":1607,"threads_started":1}
I20260807 07:03:47.633553 6358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:03:47.596473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:46927 (local address 127.21.83.62:40861)
0807 07:03:47.597862 (+ 1389us) server_negotiation.cc:206] Beginning negotiation
0807 07:03:47.597870 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 07:03:47.600993 (+ 3123us) server_negotiation.cc:407] Connection header received
0807 07:03:47.602552 (+ 1559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:03:47.602559 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:03:47.602665 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:03:47.602798 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:03:47.605475 (+ 2677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:03:47.606642 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:03:47.610058 (+ 3416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:03:47.610852 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:03:47.610982 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:03:47.619107 (+ 8125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:03:47.619116 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:03:47.619121 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:03:47.619165 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:03:47.625230 (+ 6065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:03:47.626567 (+ 1337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:03:47.626573 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:03:47.626577 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:03:47.626882 (+ 305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:03:47.627697 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:03:47.627703 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:03:47.627707 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:03:47.628206 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:03:47.628361 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0807 07:03:47.632632 (+ 4271us) server_negotiation.cc:299] Negotiation successful
0807 07:03:47.632936 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1162,"thread_start_us":864,"threads_started":1}
I20260807 07:03:47.636420 6352 heartbeater.cc:347] Connected to a master server at 127.21.83.62:40861
I20260807 07:03:47.636915 6352 heartbeater.cc:464] Registering TS with master...
I20260807 07:03:47.637851 6352 heartbeater.cc:511] Master 127.21.83.62:40861 requested a full tablet report, sending...
I20260807 07:03:47.641024 6029 ts_manager.cc:194] Registered new tserver with Master: bbf7de1958f343069d4339a56a592be5 (127.21.83.1:42823)
WARNING: no policy specified for kudu/127.21.83.2@KRBTEST.COM; defaulting to no policy
I20260807 07:03:47.643921 6029 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.1@KRBTEST.COM'} at 127.21.83.1:46927
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab.
Entry for principal kudu/127.21.83.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab.
I20260807 07:03:47.692456 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.83.20:34963/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.2:0
--local_ip_for_outbound_sockets=127.21.83.2
--webserver_interface=127.21.83.2
--webserver_port=0
--tserver_master_addrs=127.21.83.62:40861
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab
--principal=kudu/127.21.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:34227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 07:03:47 dist-test-slave-ncp9 krb5kdc[5317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086226, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for HTTP/127.21.83.20@KRBTEST.COM
W20260807 07:03:48.331265 6363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:03:48.331557 6363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:03:48.331702 6363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:03:48.339584 6363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:03:48.339718 6363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:03:48.339896 6363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.2
I20260807 07:03:48.350777 6363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:34227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.21.83.20:34963/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab
--principal=kudu/127.21.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.83.2
--webserver_port=0
--tserver_master_addrs=127.21.83.62:40861
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.2
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:03:48.352137 6363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:03:48.353515 6363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:03:48.368099 6386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:03:48.368134 6390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:03:48.368237 6387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:03:48.370277 6363 server_base.cc:1034] running on GCE node
I20260807 07:03:48.370915 6363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:03:48.372020 6363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:03:48.373243 6363 hybrid_clock.cc:648] HybridClock initialized: now 1786086228373162 us; error 60 us; skew 500 ppm
I20260807 07:03:48.373572 6363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:03:48 dist-test-slave-ncp9 krb5kdc[5317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086228, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:03:48.377544 6363 init.cc:377] Logged in from keytab as kudu/127.21.83.2@KRBTEST.COM (short username kudu)
I20260807 07:03:48.379403 6363 webserver.cc:460] Webserver started at http://127.21.83.2:37133/ using document root <none> and password file <none>
I20260807 07:03:48.380203 6363 fs_manager.cc:359] Metadata directory not provided
I20260807 07:03:48.380360 6363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:03:48.380705 6363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 07:03:48 dist-test-slave-ncp9 krb5kdc[5317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086228, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.2@KRBTEST.COM for HTTP/127.21.83.20@KRBTEST.COM
I20260807 07:03:48.399531 6363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-1/data/instance:
uuid: "655f79e168f046c6b4fa2809df65b705"
format_stamp: "Formatted at 2026-08-07 07:03:48 on dist-test-slave-ncp9"
server_key: "efc2bbcd11ed0c01db4fa0d6bc573dbc"
server_key_iv: "6588f65a07857bf3a0923dfe04d4ff73"
server_key_version: "kuduclusterkey@0"
I20260807 07:03:48.400790 6363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-1/wal/instance:
uuid: "655f79e168f046c6b4fa2809df65b705"
format_stamp: "Formatted at 2026-08-07 07:03:48 on dist-test-slave-ncp9"
server_key: "efc2bbcd11ed0c01db4fa0d6bc573dbc"
server_key_iv: "6588f65a07857bf3a0923dfe04d4ff73"
server_key_version: "kuduclusterkey@0"
I20260807 07:03:48.408166 6363 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260807 07:03:48.426198 6397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:03:48.427979 6363 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260807 07:03:48.428205 6363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-1/wal
uuid: "655f79e168f046c6b4fa2809df65b705"
format_stamp: "Formatted at 2026-08-07 07:03:48 on dist-test-slave-ncp9"
server_key: "efc2bbcd11ed0c01db4fa0d6bc573dbc"
server_key_iv: "6588f65a07857bf3a0923dfe04d4ff73"
server_key_version: "kuduclusterkey@0"
I20260807 07:03:48.428540 6363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:03:48.456534 6363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:03:48.462110 6363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:03:48.462584 6363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:03:48.463937 6363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:03:48.465997 6363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:03:48.466118 6363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:03:48.466280 6363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:03:48.466369 6363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:03:48.526444 6363 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.2:33103
I20260807 07:03:48.526465 6568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.2:33103 every 8 connection(s)
I20260807 07:03:48.528218 6363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
I20260807 07:03:48.534174 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 6363
I20260807 07:03:48.534346 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-1/wal/instance
I20260807 07:03:48.534947 21836 external_mini_cluster.cc:1428] Setting key c5e891e73bc7262bf1658afc967d1796
Aug 07 07:03:48 dist-test-slave-ncp9 krb5kdc[5317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086228, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.2@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:03:48.551644 6572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:03:48.531229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:40605 (local address 127.21.83.62:40861)
0807 07:03:48.531864 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0807 07:03:48.531869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:03:48.533074 (+ 1205us) server_negotiation.cc:407] Connection header received
0807 07:03:48.535744 (+ 2670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:03:48.535750 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:03:48.535839 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:03:48.535942 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:03:48.538322 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:03:48.539184 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:03:48.540884 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:03:48.541368 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:03:48.541481 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:03:48.545137 (+ 3656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:03:48.545142 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:03:48.545146 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:03:48.545183 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:03:48.548148 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:03:48.548968 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:03:48.548972 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:03:48.548974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:03:48.549143 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:03:48.549689 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:03:48.549695 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:03:48.549698 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:03:48.550052 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:03:48.550211 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0807 07:03:48.550977 (+ 766us) server_negotiation.cc:299] Negotiation successful
0807 07:03:48.551221 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":387,"threads_started":1}
I20260807 07:03:48.553046 6573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:03:48.531821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40861 (local address 127.21.83.2:40605)
0807 07:03:48.532802 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0807 07:03:48.532848 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 07:03:48.535325 (+ 2477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:03:48.536849 (+ 1524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:03:48.536860 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:03:48.537511 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:03:48.538154 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:03:48.538173 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:03:48.539347 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:03:48.539351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:03:48.540743 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:03:48.540756 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:03:48.541629 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:03:48.541638 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:03:48.541913 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:03:48.542582 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:03:48.542604 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:03:48.544870 (+ 2266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:03:48.548291 (+ 3421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:03:48.548301 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:03:48.548318 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:03:48.548823 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:03:48.549272 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:03:48.549277 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:03:48.549281 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:03:48.549542 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:03:48.550204 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:03:48.550213 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:03:48.550566 (+ 353us) client_negotiation.cc:769] Sending connection context
0807 07:03:48.550809 (+ 243us) client_negotiation.cc:240] Negotiation successful
0807 07:03:48.551386 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"thread_start_us":544,"threads_started":1}
I20260807 07:03:48.554831 6569 heartbeater.cc:347] Connected to a master server at 127.21.83.62:40861
I20260807 07:03:48.555226 6569 heartbeater.cc:464] Registering TS with master...
I20260807 07:03:48.556273 6569 heartbeater.cc:511] Master 127.21.83.62:40861 requested a full tablet report, sending...
I20260807 07:03:48.558152 6029 ts_manager.cc:194] Registered new tserver with Master: 655f79e168f046c6b4fa2809df65b705 (127.21.83.2:33103)
I20260807 07:03:48.559346 6029 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.2@KRBTEST.COM'} at 127.21.83.2:40605
WARNING: no policy specified for kudu/127.21.83.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab.
Entry for principal kudu/127.21.83.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab.
I20260807 07:03:48.608076 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.83.20:34963/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.3:0
--local_ip_for_outbound_sockets=127.21.83.3
--webserver_interface=127.21.83.3
--webserver_port=0
--tserver_master_addrs=127.21.83.62:40861
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab
--principal=kudu/127.21.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:34227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 07:03:48.653347 6352 heartbeater.cc:503] Master 127.21.83.62:40861 was elected leader, sending a full tablet report...
W20260807 07:03:49.083667 6577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:03:49.084035 6577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:03:49.084260 6577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:03:49.096297 6577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:03:49.096565 6577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:03:49.096812 6577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.3
I20260807 07:03:49.109218 6577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:34227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.21.83.20:34963/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab
--principal=kudu/127.21.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.83.3
--webserver_port=0
--tserver_master_addrs=127.21.83.62:40861
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.3
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:03:49.110538 6577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:03:49.111820 6577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:03:49.126189 6582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:03:49.126942 6584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:03:49.126978 6586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:03:49.127619 6577 server_base.cc:1034] running on GCE node
I20260807 07:03:49.128159 6577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:03:49.129068 6577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:03:49.130275 6577 hybrid_clock.cc:648] HybridClock initialized: now 1786086229130221 us; error 30 us; skew 500 ppm
I20260807 07:03:49.130599 6577 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:03:49 dist-test-slave-ncp9 krb5kdc[5317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086229, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:03:49.133957 6577 init.cc:377] Logged in from keytab as kudu/127.21.83.3@KRBTEST.COM (short username kudu)
I20260807 07:03:49.135423 6577 webserver.cc:460] Webserver started at http://127.21.83.3:37187/ using document root <none> and password file <none>
I20260807 07:03:49.136144 6577 fs_manager.cc:359] Metadata directory not provided
I20260807 07:03:49.136289 6577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:03:49.136621 6577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 07:03:49 dist-test-slave-ncp9 krb5kdc[5317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086229, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.3@KRBTEST.COM for HTTP/127.21.83.20@KRBTEST.COM
I20260807 07:03:49.154328 6577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-2/data/instance:
uuid: "f605bbe276664370bc87956c43f12454"
format_stamp: "Formatted at 2026-08-07 07:03:49 on dist-test-slave-ncp9"
server_key: "05d2133468c9c8da1347f71d04617bcd"
server_key_iv: "10ee36e05de32555154b14f6fa245aa2"
server_key_version: "kuduclusterkey@0"
I20260807 07:03:49.155608 6577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-2/wal/instance:
uuid: "f605bbe276664370bc87956c43f12454"
format_stamp: "Formatted at 2026-08-07 07:03:49 on dist-test-slave-ncp9"
server_key: "05d2133468c9c8da1347f71d04617bcd"
server_key_iv: "10ee36e05de32555154b14f6fa245aa2"
server_key_version: "kuduclusterkey@0"
I20260807 07:03:49.164501 6577 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.001s sys 0.009s
I20260807 07:03:49.184114 6597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:03:49.186214 6577 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260807 07:03:49.186515 6577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-2/wal
uuid: "f605bbe276664370bc87956c43f12454"
format_stamp: "Formatted at 2026-08-07 07:03:49 on dist-test-slave-ncp9"
server_key: "05d2133468c9c8da1347f71d04617bcd"
server_key_iv: "10ee36e05de32555154b14f6fa245aa2"
server_key_version: "kuduclusterkey@0"
I20260807 07:03:49.186923 6577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:03:49.228551 6577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:03:49.234038 6577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:03:49.234632 6577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:03:49.236662 6577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:03:49.239583 6577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:03:49.239770 6577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:03:49.240041 6577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:03:49.240149 6577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:03:49.301714 6577 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.3:41823
I20260807 07:03:49.301751 6758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.3:41823 every 8 connection(s)
I20260807 07:03:49.303232 6577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
I20260807 07:03:49.311368 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 6577
I20260807 07:03:49.311559 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-2/wal/instance
I20260807 07:03:49.312040 21836 external_mini_cluster.cc:1428] Setting key 2ff8391e42e3e2f0396ddd372e4b51e7
Aug 07 07:03:49 dist-test-slave-ncp9 krb5kdc[5317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086229, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.3@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:03:49.328423 6763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:03:49.306343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:35109 (local address 127.21.83.62:40861)
0807 07:03:49.307035 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0807 07:03:49.307042 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:03:49.308851 (+ 1809us) server_negotiation.cc:407] Connection header received
0807 07:03:49.311147 (+ 2296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:03:49.311154 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:03:49.311262 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:03:49.311362 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:03:49.313415 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:03:49.314327 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:03:49.316330 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:03:49.316860 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:03:49.317066 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:03:49.321341 (+ 4275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:03:49.321347 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:03:49.321350 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:03:49.321398 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:03:49.324568 (+ 3170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:03:49.325581 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:03:49.325586 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:03:49.325589 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:03:49.325787 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:03:49.326415 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:03:49.326419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:03:49.326421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:03:49.326782 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:03:49.326950 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0807 07:03:49.327712 (+ 762us) server_negotiation.cc:299] Negotiation successful
0807 07:03:49.328010 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":421,"threads_started":1}
I20260807 07:03:49.329500 6764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:03:49.307068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40861 (local address 127.21.83.3:35109)
0807 07:03:49.308628 (+ 1560us) negotiation.cc:107] Waiting for socket to connect
0807 07:03:49.308682 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0807 07:03:49.310785 (+ 2103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:03:49.311704 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:03:49.311713 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:03:49.312303 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:03:49.313239 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:03:49.313263 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:03:49.314526 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:03:49.314530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:03:49.316156 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:03:49.316171 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:03:49.317181 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:03:49.317189 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:03:49.317496 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:03:49.318175 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:03:49.318200 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:03:49.321013 (+ 2813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:03:49.324779 (+ 3766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:03:49.324788 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:03:49.324803 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:03:49.325430 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:03:49.325988 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:03:49.325994 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:03:49.325998 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:03:49.326282 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:03:49.326936 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:03:49.326945 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:03:49.327311 (+ 366us) client_negotiation.cc:769] Sending connection context
0807 07:03:49.327590 (+ 279us) client_negotiation.cc:240] Negotiation successful
0807 07:03:49.328203 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1002,"thread_start_us":660,"threads_started":1}
I20260807 07:03:49.331061 6760 heartbeater.cc:347] Connected to a master server at 127.21.83.62:40861
I20260807 07:03:49.331378 6760 heartbeater.cc:464] Registering TS with master...
I20260807 07:03:49.332232 6760 heartbeater.cc:511] Master 127.21.83.62:40861 requested a full tablet report, sending...
I20260807 07:03:49.333869 6029 ts_manager.cc:194] Registered new tserver with Master: f605bbe276664370bc87956c43f12454 (127.21.83.3:41823)
I20260807 07:03:49.334735 6029 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.3@KRBTEST.COM'} at 127.21.83.3:35109
I20260807 07:03:49.340309 21836 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 07:03:49.360021 6763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:03:49.345504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48686 (local address 127.21.83.62:40861)
0807 07:03:49.345675 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0807 07:03:49.345679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:03:49.346364 (+ 685us) server_negotiation.cc:407] Connection header received
0807 07:03:49.346531 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:03:49.346536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:03:49.346648 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:03:49.346754 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:03:49.348428 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:03:49.349346 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:03:49.350843 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:03:49.351377 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:03:49.351487 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:03:49.354090 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:03:49.354096 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:03:49.354100 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:03:49.354138 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:03:49.357337 (+ 3199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:03:49.358105 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:03:49.358110 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:03:49.358112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:03:49.358278 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:03:49.358661 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:03:49.358666 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:03:49.358668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:03:49.359015 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:03:49.359236 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0807 07:03:49.359575 (+ 339us) server_negotiation.cc:299] Negotiation successful
0807 07:03:49.359773 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 07:03:49.364468 6029 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48686:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 07:03:49.373746 6029 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 07:03:49.405699 6781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:03:49.390501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.2:33103 (local address 127.0.0.1:48998)
0807 07:03:49.395572 (+ 5071us) negotiation.cc:107] Waiting for socket to connect
0807 07:03:49.395593 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 07:03:49.395836 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:03:49.397179 (+ 1343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:03:49.397183 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:03:49.397206 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:03:49.397781 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:03:49.397797 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:03:49.399809 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:03:49.399815 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:03:49.401690 (+ 1875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:03:49.401703 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:03:49.404321 (+ 2618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:03:49.404327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:03:49.404579 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:03:49.404657 (+ 78us) client_negotiation.cc:769] Sending connection context
0807 07:03:49.404815 (+ 158us) client_negotiation.cc:240] Negotiation successful
0807 07:03:49.405089 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4911,"mutex_wait_us":22,"thread_start_us":4764,"threads_started":1}
I20260807 07:03:49.407033 6782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:03:49.391046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48998 (local address 127.21.83.2:33103)
0807 07:03:49.394293 (+ 3247us) server_negotiation.cc:206] Beginning negotiation
0807 07:03:49.394304 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 07:03:49.395707 (+ 1403us) server_negotiation.cc:407] Connection header received
0807 07:03:49.396104 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:03:49.396112 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:03:49.396390 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:03:49.396788 (+ 398us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:03:49.398028 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:03:49.399687 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:03:49.401923 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:03:49.403714 (+ 1791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:03:49.403854 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:03:49.404108 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0807 07:03:49.406123 (+ 2015us) server_negotiation.cc:299] Negotiation successful
0807 07:03:49.406443 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2914,"thread_start_us":2784,"threads_started":1}
I20260807 07:03:49.407291 6780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:03:49.390127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57054 (local address 127.21.83.3:41823)
0807 07:03:49.391722 (+ 1595us) server_negotiation.cc:206] Beginning negotiation
0807 07:03:49.391733 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 07:03:49.393030 (+ 1297us) server_negotiation.cc:407] Connection header received
0807 07:03:49.394982 (+ 1952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:03:49.394992 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:03:49.395295 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:03:49.395522 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:03:49.397026 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:03:49.398754 (+ 1728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:03:49.401856 (+ 3102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:03:49.403714 (+ 1858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:03:49.403875 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:03:49.404133 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0807 07:03:49.406658 (+ 2525us) server_negotiation.cc:299] Negotiation successful
0807 07:03:49.406878 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1325,"thread_start_us":1104,"threads_started":1}
I20260807 07:03:49.408394 6779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:03:49.389761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.3:41823 (local address 127.0.0.1:57054)
0807 07:03:49.392791 (+ 3030us) negotiation.cc:107] Waiting for socket to connect
0807 07:03:49.392843 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0807 07:03:49.394721 (+ 1878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:03:49.396053 (+ 1332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:03:49.396060 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:03:49.396090 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:03:49.396712 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:03:49.396728 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:03:49.398939 (+ 2211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:03:49.398944 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:03:49.401698 (+ 2754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:03:49.401712 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:03:49.406211 (+ 4499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:03:49.406219 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:03:49.406405 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:03:49.406442 (+ 37us) client_negotiation.cc:769] Sending connection context
0807 07:03:49.407478 (+ 1036us) client_negotiation.cc:240] Negotiation successful
0807 07:03:49.407647 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2688,"mutex_wait_us":43,"thread_start_us":2333,"threads_started":1}
I20260807 07:03:49.409312 6788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:03:49.396194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:42823 (local address 127.0.0.1:48376)
0807 07:03:49.397233 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0807 07:03:49.397264 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0807 07:03:49.397420 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:03:49.398711 (+ 1291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:03:49.398716 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:03:49.398744 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:03:49.399541 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:03:49.399555 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:03:49.401693 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:03:49.401699 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:03:49.404028 (+ 2329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:03:49.404049 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:03:49.407567 (+ 3518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:03:49.407574 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:03:49.407883 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:03:49.407916 (+ 33us) client_negotiation.cc:769] Sending connection context
0807 07:03:49.408010 (+ 94us) client_negotiation.cc:240] Negotiation successful
0807 07:03:49.408165 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":899,"thread_start_us":718,"threads_started":1}
I20260807 07:03:49.410027 6783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:03:49.394468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48376 (local address 127.21.83.1:42823)
0807 07:03:49.395912 (+ 1444us) server_negotiation.cc:206] Beginning negotiation
0807 07:03:49.395924 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 07:03:49.397417 (+ 1493us) server_negotiation.cc:407] Connection header received
0807 07:03:49.397889 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:03:49.397899 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:03:49.398186 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:03:49.398405 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:03:49.399752 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:03:49.401514 (+ 1762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:03:49.404271 (+ 2757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:03:49.406912 (+ 2641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:03:49.407050 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:03:49.407343 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0807 07:03:49.408506 (+ 1163us) server_negotiation.cc:299] Negotiation successful
0807 07:03:49.408757 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1208,"thread_start_us":1388,"threads_started":1}
I20260807 07:03:49.411731 6459 tablet_service.cc:1467] Processing CreateTablet for tablet 4b952262a52740a2aedcc44bdf1232a9 (DEFAULT_TABLE table=test-table [id=5c3cf392d9b44b8e947962bb9b561f79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:03:49.413625 6459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b952262a52740a2aedcc44bdf1232a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:03:49.416260 6255 tablet_service.cc:1467] Processing CreateTablet for tablet 4b952262a52740a2aedcc44bdf1232a9 (DEFAULT_TABLE table=test-table [id=5c3cf392d9b44b8e947962bb9b561f79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:03:49.417584 6666 tablet_service.cc:1467] Processing CreateTablet for tablet 4b952262a52740a2aedcc44bdf1232a9 (DEFAULT_TABLE table=test-table [id=5c3cf392d9b44b8e947962bb9b561f79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:03:49.418515 6255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b952262a52740a2aedcc44bdf1232a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:03:49.419538 6666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b952262a52740a2aedcc44bdf1232a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:03:49.435664 6789 tablet_bootstrap.cc:492] T 4b952262a52740a2aedcc44bdf1232a9 P 655f79e168f046c6b4fa2809df65b705: Bootstrap starting.
I20260807 07:03:49.438051 6790 tablet_bootstrap.cc:492] T 4b952262a52740a2aedcc44bdf1232a9 P bbf7de1958f343069d4339a56a592be5: Bootstrap starting.
I20260807 07:03:49.440713 6791 tablet_bootstrap.cc:492] T 4b952262a52740a2aedcc44bdf1232a9 P f605bbe276664370bc87956c43f12454: Bootstrap starting.
I20260807 07:03:49.443152 6789 tablet_bootstrap.cc:654] T 4b952262a52740a2aedcc44bdf1232a9 P 655f79e168f046c6b4fa2809df65b705: Neither blocks nor log segments found. Creating new log.
I20260807 07:03:49.443607 6790 tablet_bootstrap.cc:654] T 4b952262a52740a2aedcc44bdf1232a9 P bbf7de1958f343069d4339a56a592be5: Neither blocks nor log segments found. Creating new log.
I20260807 07:03:49.446177 6789 log.cc:824] T 4b952262a52740a2aedcc44bdf1232a9 P 655f79e168f046c6b4fa2809df65b705: Log is configured to *not* fsync() on all Append() calls
I20260807 07:03:49.446318 6790 log.cc:824] T 4b952262a52740a2aedcc44bdf1232a9 P bbf7de1958f343069d4339a56a592be5: Log is configured to *not* fsync() on all Append() calls
I20260807 07:03:49.446880 6791 tablet_bootstrap.cc:654] T 4b952262a52740a2aedcc44bdf1232a9 P f605bbe276664370bc87956c43f12454: Neither blocks nor log segments found. Creating new log.
I20260807 07:03:49.449317 6791 log.cc:824] T 4b952262a52740a2aedcc44bdf1232a9 P f605bbe276664370bc87956c43f12454: Log is configured to *not* fsync() on all Append() calls
I20260807 07:03:49.454546 6791 tablet_bootstrap.cc:492] T 4b952262a52740a2aedcc44bdf1232a9 P f605bbe276664370bc87956c43f12454: No bootstrap required, opened a new log
I20260807 07:03:49.454973 6791 ts_tablet_manager.cc:1397] T 4b952262a52740a2aedcc44bdf1232a9 P f605bbe276664370bc87956c43f12454: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.005s
I20260807 07:03:49.455546 6789 tablet_bootstrap.cc:492] T 4b952262a52740a2aedcc44bdf1232a9 P 655f79e168f046c6b4fa2809df65b705: No bootstrap required, opened a new log
I20260807 07:03:49.456182 6789 ts_tablet_manager.cc:1397] T 4b952262a52740a2aedcc44bdf1232a9 P 655f79e168f046c6b4fa2809df65b705: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.006s
I20260807 07:03:49.456260 6790 tablet_bootstrap.cc:492] T 4b952262a52740a2aedcc44bdf1232a9 P bbf7de1958f343069d4339a56a592be5: No bootstrap required, opened a new log
I20260807 07:03:49.456822 6790 ts_tablet_manager.cc:1397] T 4b952262a52740a2aedcc44bdf1232a9 P bbf7de1958f343069d4339a56a592be5: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.011s
I20260807 07:03:49.469636 6790 raft_consensus.cc:348] T 4b952262a52740a2aedcc44bdf1232a9 P bbf7de1958f343069d4339a56a592be5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f605bbe276664370bc87956c43f12454" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 41823 } } peers { permanent_uuid: "bbf7de1958f343069d4339a56a592be5" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42823 } } peers { permanent_uuid: "655f79e168f046c6b4fa2809df65b705" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 33103 } }
I20260807 07:03:49.470467 6790 raft_consensus.cc:374] T 4b952262a52740a2aedcc44bdf1232a9 P bbf7de1958f343069d4339a56a592be5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:03:49.470638 6790 raft_consensus.cc:729] T 4b952262a52740a2aedcc44bdf1232a9 P bbf7de1958f343069d4339a56a592be5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7de1958f343069d4339a56a592be5, State: Initialized, Role: FOLLOWER
I20260807 07:03:49.471405 6790 consensus_queue.cc:260] T 4b952262a52740a2aedcc44bdf1232a9 P bbf7de1958f343069d4339a56a592be5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f605bbe276664370bc87956c43f12454" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 41823 } } peers { permanent_uuid: "bbf7de1958f343069d4339a56a592be5" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42823 } } peers { permanent_uuid: "655f79e168f046c6b4fa2809df65b705" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 33103 } }
I20260807 07:03:49.472321 6791 raft_consensus.cc:348] T 4b952262a52740a2aedcc44bdf1232a9 P f605bbe276664370bc87956c43f12454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f605bbe276664370bc87956c43f12454" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 41823 } } peers { permanent_uuid: "bbf7de1958f343069d4339a56a592be5" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42823 } } peers { permanent_uuid: "655f79e168f046c6b4fa2809df65b705" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 33103 } }
I20260807 07:03:49.473024 6791 raft_consensus.cc:374] T 4b952262a52740a2aedcc44bdf1232a9 P f605bbe276664370bc87956c43f12454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:03:49.473203 6791 raft_consensus.cc:729] T 4b952262a52740a2aedcc44bdf1232a9 P f605bbe276664370bc87956c43f12454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f605bbe276664370bc87956c43f12454, State: Initialized, Role: FOLLOWER
I20260807 07:03:49.474318 6790 ts_tablet_manager.cc:1428] T 4b952262a52740a2aedcc44bdf1232a9 P bbf7de1958f343069d4339a56a592be5: Time spent starting tablet: real 0.017s user 0.015s sys 0.001s
I20260807 07:03:49.474135 6791 consensus_queue.cc:260] T 4b952262a52740a2aedcc44bdf1232a9 P f605bbe276664370bc87956c43f12454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f605bbe276664370bc87956c43f12454" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 41823 } } peers { permanent_uuid: "bbf7de1958f343069d4339a56a592be5" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42823 } } peers { permanent_uuid: "655f79e168f046c6b4fa2809df65b705" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 33103 } }
I20260807 07:03:49.474069 6789 raft_consensus.cc:348] T 4b952262a52740a2aedcc44bdf1232a9 P 655f79e168f046c6b4fa2809df65b705 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f605bbe276664370bc87956c43f12454" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 41823 } } peers { permanent_uuid: "bbf7de1958f343069d4339a56a592be5" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42823 } } peers { permanent_uuid: "655f79e168f046c6b4fa2809df65b705" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 33103 } }
I20260807 07:03:49.474855 6789 raft_consensus.cc:374] T 4b952262a52740a2aedcc44bdf1232a9 P 655f79e168f046c6b4fa2809df65b705 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:03:49.475092 6789 raft_consensus.cc:729] T 4b952262a52740a2aedcc44bdf1232a9 P 655f79e168f046c6b4fa2809df65b705 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 655f79e168f046c6b4fa2809df65b705, State: Initialized, Role: FOLLOWER
I20260807 07:03:49.476248 6789 consensus_queue.cc:260] T 4b952262a52740a2aedcc44bdf1232a9 P 655f79e168f046c6b4fa2809df65b705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f605bbe276664370bc87956c43f12454" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 41823 } } peers { permanent_uuid: "bbf7de1958f343069d4339a56a592be5" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42823 } } peers { permanent_uuid: "655f79e168f046c6b4fa2809df65b705" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 33103 } }
I20260807 07:03:49.478212 6760 heartbeater.cc:503] Master 127.21.83.62:40861 was elected leader, sending a full tablet report...
I20260807 07:03:49.479260 6569 heartbeater.cc:503] Master 127.21.83.62:40861 was elected leader, sending a full tablet report...
I20260807 07:03:49.481360 6791 ts_tablet_manager.cc:1428] T 4b952262a52740a2aedcc44bdf1232a9 P f605bbe276664370bc87956c43f12454: Time spent starting tablet: real 0.026s user 0.021s sys 0.003s
I20260807 07:03:49.482028 6789 ts_tablet_manager.cc:1428] T 4b952262a52740a2aedcc44bdf1232a9 P 655f79e168f046c6b4fa2809df65b705: Time spent starting tablet: real 0.026s user 0.025s sys 0.000s
W20260807 07:03:49.531235 6571 tablet.cc:2369] T 4b952262a52740a2aedcc44bdf1232a9 P 655f79e168f046c6b4fa2809df65b705: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 07:03:49.555928 6761 tablet.cc:2369] T 4b952262a52740a2aedcc44bdf1232a9 P f605bbe276664370bc87956c43f12454: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 07:03:49.597914 6356 tablet.cc:2369] T 4b952262a52740a2aedcc44bdf1232a9 P bbf7de1958f343069d4339a56a592be5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 07:03:49.674068 6798 raft_consensus.cc:482] T 4b952262a52740a2aedcc44bdf1232a9 P f605bbe276664370bc87956c43f12454 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:03:49.674486 6798 raft_consensus.cc:504] T 4b952262a52740a2aedcc44bdf1232a9 P f605bbe276664370bc87956c43f12454 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f605bbe276664370bc87956c43f12454" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 41823 } } peers { permanent_uuid: "bbf7de1958f343069d4339a56a592be5" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42823 } } peers { permanent_uuid: "655f79e168f046c6b4fa2809df65b705" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 33103 } }
I20260807 07:03:49.677227 6798 leader_election.cc:288] T 4b952262a52740a2aedcc44bdf1232a9 P f605bbe276664370bc87956c43f12454 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf7de1958f343069d4339a56a592be5 (127.21.83.1:42823), 655f79e168f046c6b4fa2809df65b705 (127.21.83.2:33103)
I20260807 07:03:49.688071 6764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:03:49.677284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:42823 (local address 127.21.83.3:41397)
0807 07:03:49.677518 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0807 07:03:49.677541 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 07:03:49.677869 (+ 328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:03:49.678642 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:03:49.678647 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:03:49.678694 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:03:49.678977 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:03:49.678991 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:03:49.681738 (+ 2747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:03:49.681744 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:03:49.684849 (+ 3105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:03:49.684865 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:03:49.687094 (+ 2229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:03:49.687099 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:03:49.687378 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:03:49.687413 (+ 35us) client_negotiation.cc:769] Sending connection context
0807 07:03:49.687516 (+ 103us) client_negotiation.cc:240] Negotiation successful
0807 07:03:49.687681 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"mutex_wait_us":23}
I20260807 07:03:49.688547 6783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:03:49.677792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:41397 (local address 127.21.83.1:42823)
0807 07:03:49.678129 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0807 07:03:49.678136 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:03:49.678158 (+ 22us) server_negotiation.cc:407] Connection header received
0807 07:03:49.678265 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:03:49.678288 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:03:49.678374 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:03:49.678479 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:03:49.679200 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:03:49.681594 (+ 2394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:03:49.685228 (+ 3634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:03:49.686914 (+ 1686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:03:49.687075 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:03:49.687214 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0807 07:03:49.688035 (+ 821us) server_negotiation.cc:299] Negotiation successful
0807 07:03:49.688192 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 07:03:49.690209 6283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b952262a52740a2aedcc44bdf1232a9" candidate_uuid: "f605bbe276664370bc87956c43f12454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7de1958f343069d4339a56a592be5" is_pre_election: true
I20260807 07:03:49.691255 6283 raft_consensus.cc:2454] T 4b952262a52740a2aedcc44bdf1232a9 P bbf7de1958f343069d4339a56a592be5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f605bbe276664370bc87956c43f12454 in term 0.
I20260807 07:03:49.691435 6782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:03:49.677908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:41591 (local address 127.21.83.2:33103)
0807 07:03:49.678175 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0807 07:03:49.678181 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:03:49.680796 (+ 2615us) server_negotiation.cc:407] Connection header received
0807 07:03:49.680969 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:03:49.680976 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:03:49.681123 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:03:49.681235 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:03:49.683069 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:03:49.684121 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:03:49.687914 (+ 3793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:03:49.689908 (+ 1994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:03:49.690104 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:03:49.690222 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0807 07:03:49.690929 (+ 707us) server_negotiation.cc:299] Negotiation successful
0807 07:03:49.691102 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 07:03:49.692447 6800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:03:49.678834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.2:33103 (local address 127.21.83.3:41591)
0807 07:03:49.680680 (+ 1846us) negotiation.cc:107] Waiting for socket to connect
0807 07:03:49.680697 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 07:03:49.680816 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:03:49.681469 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:03:49.681475 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:03:49.681507 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:03:49.682817 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:03:49.682836 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:03:49.684343 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:03:49.684348 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:03:49.687616 (+ 3268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:03:49.687636 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:03:49.690440 (+ 2804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:03:49.690448 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:03:49.690672 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:03:49.690714 (+ 42us) client_negotiation.cc:769] Sending connection context
0807 07:03:49.691716 (+ 1002us) client_negotiation.cc:240] Negotiation successful
0807 07:03:49.691880 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1704,"thread_start_us":1391,"threads_started":1}
I20260807 07:03:49.692312 6598 leader_election.cc:302] T 4b952262a52740a2aedcc44bdf1232a9 P f605bbe276664370bc87956c43f12454 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbf7de1958f343069d4339a56a592be5, f605bbe276664370bc87956c43f12454; no voters:
I20260807 07:03:49.693106 6798 raft_consensus.cc:2781] T 4b952262a52740a2aedcc44bdf1232a9 P f605bbe276664370bc87956c43f12454 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 07:03:49.693359 6798 raft_consensus.cc:482] T 4b952262a52740a2aedcc44bdf1232a9 P f605bbe276664370bc87956c43f12454 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:03:49.693509 6798 raft_consensus.cc:3037] T 4b952262a52740a2aedcc44bdf1232a9 P f605bbe276664370bc87956c43f12454 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:03:49.694492 6498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b952262a52740a2aedcc44bdf1232a9" candidate_uuid: "f605bbe276664370bc87956c43f12454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "655f79e168f046c6b4fa2809df65b705" is_pre_election: true
I20260807 07:03:49.695305 6498 raft_consensus.cc:2454] T 4b952262a52740a2aedcc44bdf1232a9 P 655f79e168f046c6b4fa2809df65b705 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f605bbe276664370bc87956c43f12454 in term 0.
I20260807 07:03:49.698635 6798 raft_consensus.cc:504] T 4b952262a52740a2aedcc44bdf1232a9 P f605bbe276664370bc87956c43f12454 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f605bbe276664370bc87956c43f12454" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 41823 } } peers { permanent_uuid: "bbf7de1958f343069d4339a56a592be5" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42823 } } peers { permanent_uuid: "655f79e168f046c6b4fa2809df65b705" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 33103 } }
I20260807 07:03:49.699927 6798 leader_election.cc:288] T 4b952262a52740a2aedcc44bdf1232a9 P f605bbe276664370bc87956c43f12454 [CANDIDATE]: Term 1 election: Requested vote from peers bbf7de1958f343069d4339a56a592be5 (127.21.83.1:42823), 655f79e168f046c6b4fa2809df65b705 (127.21.83.2:33103)
I20260807 07:03:49.700146 6283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b952262a52740a2aedcc44bdf1232a9" candidate_uuid: "f605bbe276664370bc87956c43f12454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7de1958f343069d4339a56a592be5"
I20260807 07:03:49.700371 6283 raft_consensus.cc:3037] T 4b952262a52740a2aedcc44bdf1232a9 P bbf7de1958f343069d4339a56a592be5 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:03:49.700961 6498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b952262a52740a2aedcc44bdf1232a9" candidate_uuid: "f605bbe276664370bc87956c43f12454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "655f79e168f046c6b4fa2809df65b705"
I20260807 07:03:49.701231 6498 raft_consensus.cc:3037] T 4b952262a52740a2aedcc44bdf1232a9 P 655f79e168f046c6b4fa2809df65b705 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:03:49.702898 6283 raft_consensus.cc:2454] T 4b952262a52740a2aedcc44bdf1232a9 P bbf7de1958f343069d4339a56a592be5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f605bbe276664370bc87956c43f12454 in term 1.
I20260807 07:03:49.703357 6598 leader_election.cc:302] T 4b952262a52740a2aedcc44bdf1232a9 P f605bbe276664370bc87956c43f12454 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbf7de1958f343069d4339a56a592be5, f605bbe276664370bc87956c43f12454; no voters:
I20260807 07:03:49.703631 6798 raft_consensus.cc:2781] T 4b952262a52740a2aedcc44bdf1232a9 P f605bbe276664370bc87956c43f12454 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:03:49.704335 6798 raft_consensus.cc:686] T 4b952262a52740a2aedcc44bdf1232a9 P f605bbe276664370bc87956c43f12454 [term 1 LEADER]: Becoming Leader. State: Replica: f605bbe276664370bc87956c43f12454, State: Running, Role: LEADER
I20260807 07:03:49.704695 6498 raft_consensus.cc:2454] T 4b952262a52740a2aedcc44bdf1232a9 P 655f79e168f046c6b4fa2809df65b705 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f605bbe276664370bc87956c43f12454 in term 1.
I20260807 07:03:49.704885 6798 consensus_queue.cc:237] T 4b952262a52740a2aedcc44bdf1232a9 P f605bbe276664370bc87956c43f12454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f605bbe276664370bc87956c43f12454" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 41823 } } peers { permanent_uuid: "bbf7de1958f343069d4339a56a592be5" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42823 } } peers { permanent_uuid: "655f79e168f046c6b4fa2809df65b705" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 33103 } }
I20260807 07:03:49.710470 6029 catalog_manager.cc:5495] T 4b952262a52740a2aedcc44bdf1232a9 P f605bbe276664370bc87956c43f12454 reported cstate change: term changed from 0 to 1, leader changed from <none> to f605bbe276664370bc87956c43f12454 (127.21.83.3). New cstate: current_term: 1 leader_uuid: "f605bbe276664370bc87956c43f12454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f605bbe276664370bc87956c43f12454" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 41823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bbf7de1958f343069d4339a56a592be5" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "655f79e168f046c6b4fa2809df65b705" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 33103 } health_report { overall_health: UNKNOWN } } }
I20260807 07:03:49.782541 6780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:03:49.776078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57070 (local address 127.21.83.3:41823)
0807 07:03:49.776276 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0807 07:03:49.776281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:03:49.776466 (+ 185us) server_negotiation.cc:407] Connection header received
0807 07:03:49.776645 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:03:49.776649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:03:49.776728 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:03:49.776864 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:03:49.777384 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:03:49.778437 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:03:49.780289 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:03:49.780872 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:03:49.781003 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:03:49.781479 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:03:49.781961 (+ 482us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:03:49.782065 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0807 07:03:49.782174 (+ 109us) server_negotiation.cc:299] Negotiation successful
0807 07:03:49.782265 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260807 07:03:49.791625 6498 raft_consensus.cc:1261] T 4b952262a52740a2aedcc44bdf1232a9 P 655f79e168f046c6b4fa2809df65b705 [term 1 FOLLOWER]: Refusing update from remote peer f605bbe276664370bc87956c43f12454: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:03:49.791788 6283 raft_consensus.cc:1261] T 4b952262a52740a2aedcc44bdf1232a9 P bbf7de1958f343069d4339a56a592be5 [term 1 FOLLOWER]: Refusing update from remote peer f605bbe276664370bc87956c43f12454: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:03:49.792665 6801 consensus_queue.cc:1035] T 4b952262a52740a2aedcc44bdf1232a9 P f605bbe276664370bc87956c43f12454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655f79e168f046c6b4fa2809df65b705" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 33103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:03:49.793342 6798 consensus_queue.cc:1035] T 4b952262a52740a2aedcc44bdf1232a9 P f605bbe276664370bc87956c43f12454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7de1958f343069d4339a56a592be5" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 42823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:03:49.811113 6804 mvcc.cc:205] Tried to move back new op lower bound from 7315809197216358400 to 7315809196875411456. Current Snapshot: MvccSnapshot[applied={T|T < 7315809197216358400}]
I20260807 07:03:49.830538 6029 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:03:49.830956 6029 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:03:49.835371 6029 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 179f918925194d11b0c9c8505bac2789: Sending DeleteTablet for 3 replicas of tablet 4b952262a52740a2aedcc44bdf1232a9
I20260807 07:03:49.837088 6255 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b952262a52740a2aedcc44bdf1232a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:03:49 UTC) from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:48376
I20260807 07:03:49.837239 6666 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b952262a52740a2aedcc44bdf1232a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:03:49 UTC) from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:57054
I20260807 07:03:49.836833 6459 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b952262a52740a2aedcc44bdf1232a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:03:49 UTC) from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:48998
I20260807 07:03:49.838837 6820 tablet_replica.cc:330] T 4b952262a52740a2aedcc44bdf1232a9 P f605bbe276664370bc87956c43f12454: stopping tablet replica
I20260807 07:03:49.839524 6820 raft_consensus.cc:2229] T 4b952262a52740a2aedcc44bdf1232a9 P f605bbe276664370bc87956c43f12454 [term 1 LEADER]: Raft consensus shutting down.
I20260807 07:03:49.840519 6820 raft_consensus.cc:2258] T 4b952262a52740a2aedcc44bdf1232a9 P f605bbe276664370bc87956c43f12454 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:03:49.840930 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 6154
I20260807 07:03:49.843143 6820 ts_tablet_manager.cc:1905] T 4b952262a52740a2aedcc44bdf1232a9 P f605bbe276664370bc87956c43f12454: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:03:49.848711 6819 tablet_replica.cc:330] T 4b952262a52740a2aedcc44bdf1232a9 P bbf7de1958f343069d4339a56a592be5: stopping tablet replica
I20260807 07:03:49.849929 6819 raft_consensus.cc:2229] T 4b952262a52740a2aedcc44bdf1232a9 P bbf7de1958f343069d4339a56a592be5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:03:49.850862 6819 raft_consensus.cc:2258] T 4b952262a52740a2aedcc44bdf1232a9 P bbf7de1958f343069d4339a56a592be5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:03:49.851500 6821 tablet_replica.cc:330] T 4b952262a52740a2aedcc44bdf1232a9 P 655f79e168f046c6b4fa2809df65b705: stopping tablet replica
I20260807 07:03:49.851986 6821 raft_consensus.cc:2229] T 4b952262a52740a2aedcc44bdf1232a9 P 655f79e168f046c6b4fa2809df65b705 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:03:49.852674 6821 raft_consensus.cc:2258] T 4b952262a52740a2aedcc44bdf1232a9 P 655f79e168f046c6b4fa2809df65b705 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:03:49.853225 6819 ts_tablet_manager.cc:1905] T 4b952262a52740a2aedcc44bdf1232a9 P bbf7de1958f343069d4339a56a592be5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:03:49.853550 6820 ts_tablet_manager.cc:1918] T 4b952262a52740a2aedcc44bdf1232a9 P f605bbe276664370bc87956c43f12454: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:03:49.853739 6820 log.cc:1196] T 4b952262a52740a2aedcc44bdf1232a9 P f605bbe276664370bc87956c43f12454: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-2/wal/wals/4b952262a52740a2aedcc44bdf1232a9
I20260807 07:03:49.854334 6820 ts_tablet_manager.cc:1939] T 4b952262a52740a2aedcc44bdf1232a9 P f605bbe276664370bc87956c43f12454: Deleting consensus metadata
I20260807 07:03:49.855083 6821 ts_tablet_manager.cc:1905] T 4b952262a52740a2aedcc44bdf1232a9 P 655f79e168f046c6b4fa2809df65b705: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:03:49.856431 6004 catalog_manager.cc:4841] TS f605bbe276664370bc87956c43f12454 (127.21.83.3:41823): tablet 4b952262a52740a2aedcc44bdf1232a9 (table test-table [id=5c3cf392d9b44b8e947962bb9b561f79]) successfully deleted
I20260807 07:03:49.862582 6819 ts_tablet_manager.cc:1918] T 4b952262a52740a2aedcc44bdf1232a9 P bbf7de1958f343069d4339a56a592be5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:03:49.862931 6819 log.cc:1196] T 4b952262a52740a2aedcc44bdf1232a9 P bbf7de1958f343069d4339a56a592be5: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-0/wal/wals/4b952262a52740a2aedcc44bdf1232a9
I20260807 07:03:49.863536 6819 ts_tablet_manager.cc:1939] T 4b952262a52740a2aedcc44bdf1232a9 P bbf7de1958f343069d4339a56a592be5: Deleting consensus metadata
I20260807 07:03:49.865711 6004 catalog_manager.cc:4841] TS bbf7de1958f343069d4339a56a592be5 (127.21.83.1:42823): tablet 4b952262a52740a2aedcc44bdf1232a9 (table test-table [id=5c3cf392d9b44b8e947962bb9b561f79]) successfully deleted
I20260807 07:03:49.866974 6821 ts_tablet_manager.cc:1918] T 4b952262a52740a2aedcc44bdf1232a9 P 655f79e168f046c6b4fa2809df65b705: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Aug 07 07:03:49 dist-test-slave-ncp9 krb5kdc[5317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.1@KRBTEST.COM
I20260807 07:03:49.867271 6821 log.cc:1196] T 4b952262a52740a2aedcc44bdf1232a9 P 655f79e168f046c6b4fa2809df65b705: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786086078080656-21836-0/minicluster-data/ts-1/wal/wals/4b952262a52740a2aedcc44bdf1232a9
I20260807 07:03:49.868180 6821 ts_tablet_manager.cc:1939] T 4b952262a52740a2aedcc44bdf1232a9 P 655f79e168f046c6b4fa2809df65b705: Deleting consensus metadata
I20260807 07:03:49.870481 6005 catalog_manager.cc:4841] TS 655f79e168f046c6b4fa2809df65b705 (127.21.83.2:33103): tablet 4b952262a52740a2aedcc44bdf1232a9 (table test-table [id=5c3cf392d9b44b8e947962bb9b561f79]) successfully deleted
I20260807 07:03:49.876104 6783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:03:49.841685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48392 (local address 127.21.83.1:42823)
0807 07:03:49.841867 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0807 07:03:49.841874 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:03:49.849825 (+ 7951us) server_negotiation.cc:407] Connection header received
0807 07:03:49.849988 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:03:49.849994 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:03:49.850158 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:03:49.850268 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:03:49.856120 (+ 5852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:03:49.857307 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:03:49.859388 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:03:49.860180 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:03:49.860291 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:03:49.868210 (+ 7919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:03:49.868224 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:03:49.868238 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:03:49.868284 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:03:49.872867 (+ 4583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:03:49.873626 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:03:49.873634 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:03:49.873636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:03:49.873831 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:03:49.874225 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:03:49.874228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:03:49.874230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:03:49.874757 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:03:49.875226 (+ 469us) server_negotiation.cc:1035] Waiting for connection context
0807 07:03:49.875405 (+ 179us) server_negotiation.cc:299] Negotiation successful
0807 07:03:49.875660 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 07:03:49.876878 6344 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:03:49.978529 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 6154
I20260807 07:03:49.999627 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 6363
Aug 07 07:03:50 dist-test-slave-ncp9 krb5kdc[5317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.2@KRBTEST.COM
I20260807 07:03:50.019209 6782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:03:50.000468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49004 (local address 127.21.83.2:33103)
0807 07:03:50.000686 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0807 07:03:50.000691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:03:50.000713 (+ 22us) server_negotiation.cc:407] Connection header received
0807 07:03:50.000955 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:03:50.000959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:03:50.001033 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:03:50.001120 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:03:50.002534 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:03:50.003270 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:03:50.004622 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:03:50.005252 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:03:50.005377 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:03:50.011305 (+ 5928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:03:50.011330 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:03:50.011352 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:03:50.011413 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:03:50.015923 (+ 4510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:03:50.016855 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:03:50.016865 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:03:50.016869 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:03:50.017093 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:03:50.017461 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:03:50.017466 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:03:50.017470 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:03:50.017967 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:03:50.018291 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0807 07:03:50.018569 (+ 278us) server_negotiation.cc:299] Negotiation successful
0807 07:03:50.018811 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260807 07:03:50.019925 6562 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:03:50.111552 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 6363
I20260807 07:03:50.133755 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 6577
Aug 07 07:03:50 dist-test-slave-ncp9 krb5kdc[5317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.3@KRBTEST.COM
I20260807 07:03:50.150393 6780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:03:50.134482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57084 (local address 127.21.83.3:41823)
0807 07:03:50.134661 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0807 07:03:50.134667 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:03:50.134761 (+ 94us) server_negotiation.cc:407] Connection header received
0807 07:03:50.134940 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:03:50.134944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:03:50.135019 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:03:50.135113 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:03:50.136384 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:03:50.137324 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:03:50.138625 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:03:50.139090 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:03:50.139218 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:03:50.144054 (+ 4836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:03:50.144067 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:03:50.144082 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:03:50.144130 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:03:50.147295 (+ 3165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:03:50.148272 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:03:50.148280 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:03:50.148282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:03:50.148457 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:03:50.148914 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:03:50.148919 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:03:50.148922 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:03:50.149328 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:03:50.149582 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0807 07:03:50.149922 (+ 340us) server_negotiation.cc:299] Negotiation successful
0807 07:03:50.150102 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 07:03:50.151193 6751 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:03:50.271176 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 6577
I20260807 07:03:50.296247 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 5982
I20260807 07:03:50.297364 6077 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:03:50.411435 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 5982
I20260807 07:03:50.441849 21836 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260807 07:03:50.787096 21836 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260807 07:03:50.787358 21836 mini_ranger.cc:66] Stopping Ranger...
I20260807 07:03:51.143170 21836 mini_ranger.cc:68] Stopped Ranger
2026-08-07 07:03:51.143 UTC [5359] LOG: received smart shutdown request
2026-08-07 07:03:51.147 UTC [5359] LOG: background worker "logical replication launcher" (PID 5373) exited with exit code 1
2026-08-07 07:03:51.147 UTC [5364] LOG: shutting down
2026-08-07 07:03:51.277 UTC [5359] LOG: database system is shut down
2026-08-07T07:03:51Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50977 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 07:03:51 dist-test-slave-ncp9 krb5kdc[6839](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 07:03:51 dist-test-slave-ncp9 krb5kdc[6839](info): set up 2 sockets
Aug 07 07:03:51 dist-test-slave-ncp9 krb5kdc[6839](info): commencing operation
krb5kdc: starting...
W20260807 07:03:52.626957 21836 mini_kdc.cc:121] Time spent starting KDC: real 1.189s user 0.000s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Aug 07 07:03:52 dist-test-slave-ncp9 krb5kdc[6839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T07:03:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:03:52Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.21.83.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.21.83.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.21.83.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/rangeradmin_127.21.83.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.21.83.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/rangeradmin_127.21.83.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.21.83.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.21.83.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.21.83.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/rangerlookup_127.21.83.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.21.83.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/rangerlookup_127.21.83.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.83.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.21.83.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.83.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/HTTP_127.21.83.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.83.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/HTTP_127.21.83.20@KRBTEST.COM.keytab.
I20260807 07:03:53.036942 21836 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskyUa4Vb/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/postgres -l logfile start
2026-08-07 07:03:55.326 UTC [6879] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-07 07:03:55.326 UTC [6879] LOG: listening on IPv4 address "127.21.83.20", port 52599
2026-08-07 07:03:55.330 UTC [6879] LOG: listening on Unix socket "/tmp/.s.PGSQL.52599"
2026-08-07 07:03:55.352 UTC [6881] LOG: database system was shut down at 2026-08-07 07:03:54 UTC
2026-08-07 07:03:55.356 UTC [6879] LOG: database system is ready to accept connections
I20260807 07:03:56.432401 21836 mini_postgres.cc:96] Postgres bound to 52599
2026-08-07 07:03:56.458 UTC [6893] FATAL: database "slave" does not exist
127.21.83.20:52599 - accepting connections
I20260807 07:03:56.458793 21836 mini_ranger.cc:161] Starting Ranger...
I20260807 07:03:56.492694 21836 mini_ranger.cc:84] Created miniranger Postgres user
I20260807 07:03:56.941159 21836 mini_ranger.cc:87] Created ranger Postgres database
I20260807 07:03:56.941258 21836 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ranger-admin
2026-08-07 07:03:57,245 [I] DB FLAVOR :POSTGRES
2026-08-07 07:03:57,245 [I] --------- Verifying Ranger DB connection ---------
2026-08-07 07:03:57,245 [I] Checking connection..
2026-08-07 07:03:57,245 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyUa4Vb/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyUa4Vb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.83.20:52599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-07 07:03:57,491 [I] Checking connection passed.
2026-08-07 07:03:57,491 [I] --------- Verifying version history table ---------
2026-08-07 07:03:57,491 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyUa4Vb/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyUa4Vb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.83.20:52599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 07:03:57,761 [I] Table x_db_version_h does not exist in database ranger
2026-08-07 07:03:57,762 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyUa4Vb/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyUa4Vb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.83.20:52599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 07:03:58,166 [I] Table x_db_version_h does not exist in database ranger
2026-08-07 07:03:58,167 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-07 07:03:58,167 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyUa4Vb/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyUa4Vb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.83.20:52599/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyUa4Vb/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-07 07:03:58.615 UTC [6980] WARNING: there is no transaction in progress
2026-08-07 07:03:58,628 [I] create_dbversion_catalog.sql file imported successfully
2026-08-07 07:03:58,629 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyUa4Vb/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyUa4Vb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.83.20:52599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 07:03:58,921 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-07 07:03:58,922 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-07 07:03:58,922 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyUa4Vb/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyUa4Vb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.83.20:52599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-07 07:03:59,167 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyUa4Vb/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyUa4Vb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.83.20:52599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-07 07:03:59,430 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyUa4Vb/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyUa4Vb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.83.20:52599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-ncp9.c.gcp-upstream.internal','N') ;"
2026-08-07 07:03:59,676 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyUa4Vb/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyUa4Vb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.83.20:52599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-07 07:03:59,949 [I] Table x_portal_user does not exist in database ranger
2026-08-07 07:03:59,949 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyUa4Vb/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyUa4Vb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.83.20:52599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-07 07:04:00,205 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-07 07:04:00,205 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyUa4Vb/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyUa4Vb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.83.20:52599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-07 07:04:00,468 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyUa4Vb/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyUa4Vb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.83.20:52599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-07 07:04:00,716 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-07 07:04:00,717 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyUa4Vb/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyUa4Vb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.83.20:52599/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyUa4Vb/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-07 07:04:01.618 UTC [7130] WARNING: there is no transaction in progress
2026-08-07 07:04:01.634 UTC [7130] WARNING: there is no transaction in progress
2026-08-07 07:04:01.648 UTC [7130] WARNING: there is no transaction in progress
2026-08-07 07:04:01.663 UTC [7130] WARNING: there is no transaction in progress
2026-08-07 07:04:01.678 UTC [7130] WARNING: there is no transaction in progress
2026-08-07 07:04:01.695 UTC [7130] WARNING: there is no transaction in progress
2026-08-07 07:04:01.749 UTC [7130] WARNING: there is no transaction in progress
2026-08-07 07:04:01.755 UTC [7130] WARNING: there is no transaction in progress
2026-08-07 07:04:01.770 UTC [7130] WARNING: there is no transaction in progress
2026-08-07 07:04:01.781 UTC [7130] WARNING: there is no transaction in progress
2026-08-07 07:04:01.791 UTC [7130] WARNING: there is no transaction in progress
2026-08-07 07:04:01.805 UTC [7130] WARNING: there is no transaction in progress
2026-08-07 07:04:01.814 UTC [7130] WARNING: there is no transaction in progress
2026-08-07 07:04:01.821 UTC [7130] WARNING: there is no transaction in progress
2026-08-07 07:04:01.828 UTC [7130] WARNING: there is no transaction in progress
2026-08-07 07:04:01.835 UTC [7130] WARNING: there is no transaction in progress
2026-08-07 07:04:02,469 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-07 07:04:02,469 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyUa4Vb/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyUa4Vb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.83.20:52599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-07 07:04:02,734 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-07 07:04:02,734 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyUa4Vb/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyUa4Vb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.83.20:52599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-07 07:04:03,011 [I] Table x_portal_user already exists in database 'ranger'
2026-08-07 07:04:03,012 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyUa4Vb/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyUa4Vb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.83.20:52599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-07 07:04:03,282 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-07 07:04:03,283 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyUa4Vb/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyUa4Vb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.83.20:52599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-07 07:04:03,546 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyUa4Vb/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyUa4Vb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.83.20:52599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-07 07:04:03,813 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyUa4Vb/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyUa4Vb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.83.20:52599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-ncp9.c.gcp-upstream.internal';"
2026-08-07 07:04:04,067 [I] CORE_DB_SCHEMA import status has been updated
2026-08-07 07:04:04,068 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyUa4Vb/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyUa4Vb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.83.20:52599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-07 07:04:04,309 [I] DB_PATCHES have already been applied
I20260807 07:04:04.315572 21836 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ranger-admin:/tmp/dist-test-taskyUa4Vb/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskyUa4Vb/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskyUa4Vb/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyUa4Vb/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260807 07:04:04.315652 21836 mini_ranger.cc:193] Using host: 127.21.83.20
I20260807 07:04:04.340634 21836 mini_ranger.cc:227] Ranger admin URL: http://127.21.83.20:56841
Aug 07, 2026 7:04:04 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 07, 2026 7:04:04 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 07, 2026 7:04:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ranger-admin/ews/webapp
Aug 07, 2026 7:04:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 07, 2026 7:04:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 07, 2026 7:04:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ranger-admin/ews/webapp].
Aug 07, 2026 7:04:05 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260807 07:04:05.550248 21836 mini_ranger.cc:160] Time spent starting Ranger: real 9.091s user 0.003s sys 0.070s
Aug 07, 2026 7:04:26 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 07, 2026 7:04:26 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 07, 2026 7:04:26 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 07, 2026 7:04:26 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 07, 2026 7:04:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 07, 2026 7:04:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 07, 2026 7:04:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 07, 2026 7:04:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 07, 2026 7:04:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 07, 2026 7:04:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 07, 2026 7:04:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 07, 2026 7:04:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 07, 2026 7:04:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 07, 2026 7:04:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 07, 2026 7:04:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 07, 2026 7:04:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 07, 2026 7:04:26 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 07, 2026 7:04:27 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260807 07:04:28.031904 21836 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.21.83.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.21.83.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.21.83.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/rangerkms_127.21.83.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.21.83.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/rangerkms_127.21.83.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.83.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.21.83.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.83.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/HTTP_127.21.83.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.83.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/HTTP_127.21.83.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 07 07:04:28 dist-test-slave-ncp9 krb5kdc[6839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086268, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260807 07:04:28.249258 21836 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260807 07:04:28.272403 21836 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260807 07:04:28.983242 21836 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260807 07:04:28.983364 21836 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ranger-kms
I20260807 07:04:28.983403 21836 mini_ranger_kms.cc:227] Using postgres at 127.21.83.20:52599
2026-08-07 07:04:29,358 [I] DB FLAVOR :POSTGRES
2026-08-07 07:04:29,359 [I] --------- Verifying Ranger DB connection ---------
2026-08-07 07:04:29,359 [I] Checking connection
2026-08-07 07:04:29,359 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyUa4Vb/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.83.20:52599/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-07 07:04:29,625 [I] connection success
2026-08-07 07:04:29,625 [I] --------- Verifying Ranger DB tables ---------
2026-08-07 07:04:29,625 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-07 07:04:29,625 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyUa4Vb/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.83.20:52599/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-07 07:04:29,868 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-07 07:04:29,869 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-07 07:04:29,869 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-07 07:04:29,869 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyUa4Vb/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.83.20:52599/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-07 07:04:30,130 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260807 07:04:30.576014 21836 mini_ranger_kms.cc:310] Created kms service
I20260807 07:04:30.787357 21836 mini_ranger_kms.cc:326] Created kudu user
I20260807 07:04:30.829269 21836 mini_ranger_kms.cc:343] Created rangerkms user
Aug 07 07:04:30 dist-test-slave-ncp9 krb5kdc[6839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086268, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.21.83.20@KRBTEST.COM
I20260807 07:04:31.237000 21836 mini_ranger_kms.cc:384] Added ranger policy
I20260807 07:04:31.237344 21836 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ranger-kms:/tmp/dist-test-taskyUa4Vb/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyUa4Vb/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskyUa4Vb/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260807 07:04:31.237404 21836 mini_ranger_kms.cc:240] Using host: 127.21.83.20
I20260807 07:04:31.260762 21836 mini_ranger_kms.cc:276] Ranger KMS PID: 7375
I20260807 07:04:31.260993 21836 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.21.83.20:40945
Aug 07, 2026 7:04:32 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 07, 2026 7:04:32 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 07, 2026 7:04:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 07, 2026 7:04:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 07, 2026 7:04:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ranger-kms/ews/webapp].
Aug 07, 2026 7:04:32 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 07 07:04:33 dist-test-slave-ncp9 krb5kdc[6839](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786086273, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.83.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 07 07:04:33 dist-test-slave-ncp9 krb5kdc[6839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086273, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.83.20@KRBTEST.COM for HTTP/127.21.83.20@KRBTEST.COM
I20260807 07:04:33.794796 21836 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.546s user 0.081s sys 0.335s
I20260807 07:04:33.795074 21836 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260807 07:04:33.795233 21836 mini_ranger_kms.cc:401] 127.21.83.20:40945/kms/v1/keys
Aug 07 07:04:34 dist-test-slave-ncp9 krb5kdc[6839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086273, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.83.20@KRBTEST.COM for HTTP/127.21.83.20@KRBTEST.COM
Aug 07 07:04:36 dist-test-slave-ncp9 krb5kdc[6839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.21.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
I20260807 07:04:36.473053 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:38421
--webserver_interface=127.21.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:43085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:38421
--ranger_config_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.83.20:40945/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:04:36.819068 7437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:04:36.819412 7437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:04:36.819541 7437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:04:36.827343 7437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:04:36.827574 7437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:04:36.827643 7437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:04:36.827705 7437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:04:36.827765 7437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:04:36.838431 7437 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:43085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/master-0/wal
--ranger_kms_url=127.21.83.20:40945/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:38421
--ranger_config_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:38421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:04:36.839702 7437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:04:36.841104 7437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:04:36.853936 7445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:04:36.853957 7446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:04:36.854758 7437 server_base.cc:1034] running on GCE node
W20260807 07:04:36.854874 7448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:04:36.855674 7437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:04:36.857095 7437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:04:36.858318 7437 hybrid_clock.cc:648] HybridClock initialized: now 1786086276858267 us; error 42 us; skew 500 ppm
I20260807 07:04:36.858660 7437 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:04:36 dist-test-slave-ncp9 krb5kdc[6839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086276, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:04:36.861922 7437 init.cc:377] Logged in from keytab as kudu/127.21.83.62@KRBTEST.COM (short username kudu)
I20260807 07:04:36.863453 7437 webserver.cc:460] Webserver started at http://127.21.83.62:36457/ using document root <none> and password file <none>
I20260807 07:04:36.864061 7437 fs_manager.cc:359] Metadata directory not provided
I20260807 07:04:36.864173 7437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:04:36.864461 7437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 07:04:36 dist-test-slave-ncp9 krb5kdc[6839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086276, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for HTTP/127.21.83.20@KRBTEST.COM
I20260807 07:04:36.931162 7437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "41e179b646cc45b381d9798170401005"
format_stamp: "Formatted at 2026-08-07 07:04:36 on dist-test-slave-ncp9"
server_key: "1fd371de1e352bcde1dddb3b66c7acd9"
server_key_iv: "1eb18f18ccc0f4ace2ca69a790d5ea0d"
server_key_version: "kuduclusterkey@0"
I20260807 07:04:36.932296 7437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "41e179b646cc45b381d9798170401005"
format_stamp: "Formatted at 2026-08-07 07:04:36 on dist-test-slave-ncp9"
server_key: "1fd371de1e352bcde1dddb3b66c7acd9"
server_key_iv: "1eb18f18ccc0f4ace2ca69a790d5ea0d"
server_key_version: "kuduclusterkey@0"
I20260807 07:04:36.939268 7437 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260807 07:04:36.958752 7458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:04:36.960407 7437 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 07:04:36.960601 7437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "41e179b646cc45b381d9798170401005"
format_stamp: "Formatted at 2026-08-07 07:04:36 on dist-test-slave-ncp9"
server_key: "1fd371de1e352bcde1dddb3b66c7acd9"
server_key_iv: "1eb18f18ccc0f4ace2ca69a790d5ea0d"
server_key_version: "kuduclusterkey@0"
I20260807 07:04:36.960842 7437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:04:36.990275 7437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:04:36.995213 7437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:04:36.995581 7437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:04:37.028065 7437 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.62:38421
I20260807 07:04:37.028126 7531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.62:38421 every 8 connection(s)
I20260807 07:04:37.029454 7437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:04:37.033981 7532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:04:37.035650 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 7437
I20260807 07:04:37.035858 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:04:37.036533 21836 external_mini_cluster.cc:1428] Setting key 35f95bf4341f01e7cbf7f1114ced86f3
I20260807 07:04:37.048702 7532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41e179b646cc45b381d9798170401005: Bootstrap starting.
I20260807 07:04:37.053721 7532 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41e179b646cc45b381d9798170401005: Neither blocks nor log segments found. Creating new log.
I20260807 07:04:37.055128 7532 log.cc:824] T 00000000000000000000000000000000 P 41e179b646cc45b381d9798170401005: Log is configured to *not* fsync() on all Append() calls
I20260807 07:04:37.058687 7532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41e179b646cc45b381d9798170401005: No bootstrap required, opened a new log
Aug 07 07:04:37 dist-test-slave-ncp9 krb5kdc[6839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:04:37.068759 7535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:37.038663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41534 (local address 127.21.83.62:38421)
0807 07:04:37.049716 (+ 11053us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:37.049728 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:37.049769 (+ 41us) server_negotiation.cc:407] Connection header received
0807 07:04:37.051538 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:37.051570 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:37.051942 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:37.052280 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:04:37.053849 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:37.055209 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:37.056828 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:37.057520 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:37.057725 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:37.061554 (+ 3829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:04:37.061566 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:04:37.061584 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:04:37.061620 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:04:37.064676 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:37.065382 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:37.065389 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:37.065394 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:37.065541 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:37.065931 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:37.065935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:37.065937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:37.066535 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:04:37.066871 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:37.067342 (+ 471us) server_negotiation.cc:299] Negotiation successful
0807 07:04:37.067771 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10830,"thread_start_us":10598,"threads_started":1}
I20260807 07:04:37.070014 7532 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41e179b646cc45b381d9798170401005 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41e179b646cc45b381d9798170401005" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 38421 } }
I20260807 07:04:37.070492 7532 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41e179b646cc45b381d9798170401005 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:04:37.070746 7532 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41e179b646cc45b381d9798170401005 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41e179b646cc45b381d9798170401005, State: Initialized, Role: FOLLOWER
I20260807 07:04:37.071451 7532 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41e179b646cc45b381d9798170401005 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41e179b646cc45b381d9798170401005" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 38421 } }
I20260807 07:04:37.071791 7532 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41e179b646cc45b381d9798170401005 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:04:37.071996 7532 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41e179b646cc45b381d9798170401005 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:04:37.072171 7532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41e179b646cc45b381d9798170401005 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:04:37.075326 7532 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41e179b646cc45b381d9798170401005 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41e179b646cc45b381d9798170401005" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 38421 } }
I20260807 07:04:37.075875 7532 leader_election.cc:302] T 00000000000000000000000000000000 P 41e179b646cc45b381d9798170401005 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41e179b646cc45b381d9798170401005; no voters:
I20260807 07:04:37.076525 7532 leader_election.cc:288] T 00000000000000000000000000000000 P 41e179b646cc45b381d9798170401005 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 07:04:37.076610 7537 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41e179b646cc45b381d9798170401005 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:04:37.078603 7532 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41e179b646cc45b381d9798170401005 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:04:37.081161 7537 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41e179b646cc45b381d9798170401005 [term 1 LEADER]: Becoming Leader. State: Replica: 41e179b646cc45b381d9798170401005, State: Running, Role: LEADER
I20260807 07:04:37.081792 7537 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41e179b646cc45b381d9798170401005 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41e179b646cc45b381d9798170401005" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 38421 } }
I20260807 07:04:37.086437 7538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41e179b646cc45b381d9798170401005 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41e179b646cc45b381d9798170401005" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41e179b646cc45b381d9798170401005" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 38421 } } }
I20260807 07:04:37.086416 7539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41e179b646cc45b381d9798170401005 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41e179b646cc45b381d9798170401005. Latest consensus state: current_term: 1 leader_uuid: "41e179b646cc45b381d9798170401005" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41e179b646cc45b381d9798170401005" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 38421 } } }
I20260807 07:04:37.086827 7538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41e179b646cc45b381d9798170401005 [sys.catalog]: This master's current role is: LEADER
I20260807 07:04:37.086828 7539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41e179b646cc45b381d9798170401005 [sys.catalog]: This master's current role is: LEADER
I20260807 07:04:37.095831 7532 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260807 07:04:37.098807 7546 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:04:37.103438 7546 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:04:37.129908 7546 catalog_manager.cc:1347] Generated new cluster ID: 5db07495b2694d91a7dbc400bc5b80e2
I20260807 07:04:37.130040 7546 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:04:37.145505 7546 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:04:37.147329 7546 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:04:37.162314 7546 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41e179b646cc45b381d9798170401005: Generated new TSK 0
I20260807 07:04:37.163394 7546 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.83.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab.
Entry for principal kudu/127.21.83.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab.
I20260807 07:04:38.094846 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.83.20:40945/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.1:0
--local_ip_for_outbound_sockets=127.21.83.1
--webserver_interface=127.21.83.1
--webserver_port=0
--tserver_master_addrs=127.21.83.62:38421
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab
--principal=kudu/127.21.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:43085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 07:04:38 dist-test-slave-ncp9 krb5kdc[6839](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786086278, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260807 07:04:38.533360 7599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:04:38.533710 7599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:04:38.533910 7599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:04:38.545562 7599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:04:38.545771 7599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:04:38.546000 7599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.1
I20260807 07:04:38.558815 7599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:43085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.21.83.20:40945/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab
--principal=kudu/127.21.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.83.1
--webserver_port=0
--tserver_master_addrs=127.21.83.62:38421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.1
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:04:38.560411 7599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:04:38.562091 7599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:04:38.584569 7619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:04:38.584815 7618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:04:38.585273 7621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:04:38.585685 7599 server_base.cc:1034] running on GCE node
I20260807 07:04:38.586289 7599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:04:38.587239 7599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:04:38.588487 7599 hybrid_clock.cc:648] HybridClock initialized: now 1786086278588402 us; error 77 us; skew 500 ppm
I20260807 07:04:38.588811 7599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:04:38 dist-test-slave-ncp9 krb5kdc[6839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086278, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:04:38.591764 7599 init.cc:377] Logged in from keytab as kudu/127.21.83.1@KRBTEST.COM (short username kudu)
I20260807 07:04:38.593302 7599 webserver.cc:460] Webserver started at http://127.21.83.1:40549/ using document root <none> and password file <none>
I20260807 07:04:38.594017 7599 fs_manager.cc:359] Metadata directory not provided
I20260807 07:04:38.594139 7599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:04:38.594427 7599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 07:04:38 dist-test-slave-ncp9 krb5kdc[6839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086278, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.1@KRBTEST.COM for HTTP/127.21.83.20@KRBTEST.COM
I20260807 07:04:38.622771 7599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-0/data/instance:
uuid: "e25131831d6a4c2aa8d120b6d65ff5dc"
format_stamp: "Formatted at 2026-08-07 07:04:38 on dist-test-slave-ncp9"
server_key: "ffc8e1212f57b3c35f8f4b1a210d7a5e"
server_key_iv: "af71529e831f9230a3cb0594c3d6a7d5"
server_key_version: "kuduclusterkey@0"
I20260807 07:04:38.623932 7599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-0/wal/instance:
uuid: "e25131831d6a4c2aa8d120b6d65ff5dc"
format_stamp: "Formatted at 2026-08-07 07:04:38 on dist-test-slave-ncp9"
server_key: "ffc8e1212f57b3c35f8f4b1a210d7a5e"
server_key_iv: "af71529e831f9230a3cb0594c3d6a7d5"
server_key_version: "kuduclusterkey@0"
I20260807 07:04:38.642915 7599 fs_manager.cc:691] Time spent creating directory manager: real 0.019s user 0.007s sys 0.011s
I20260807 07:04:38.673523 7628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:04:38.678272 7599 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.010s sys 0.001s
I20260807 07:04:38.678539 7599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-0/wal
uuid: "e25131831d6a4c2aa8d120b6d65ff5dc"
format_stamp: "Formatted at 2026-08-07 07:04:38 on dist-test-slave-ncp9"
server_key: "ffc8e1212f57b3c35f8f4b1a210d7a5e"
server_key_iv: "af71529e831f9230a3cb0594c3d6a7d5"
server_key_version: "kuduclusterkey@0"
I20260807 07:04:38.678890 7599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:04:38.753314 7599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:04:38.758402 7599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:04:38.758780 7599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:04:38.760603 7599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:04:38.762363 7599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:04:38.762509 7599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260807 07:04:38.762682 7599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:04:38.762784 7599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:04:38.809171 7599 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.1:45467
I20260807 07:04:38.809212 7782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.1:45467 every 8 connection(s)
Aug 07 07:04:38 dist-test-slave-ncp9 krb5kdc[6839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086278, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for HTTP/127.21.83.20@KRBTEST.COM
I20260807 07:04:38.810803 7599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
I20260807 07:04:38.813310 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 7599
I20260807 07:04:38.813550 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-0/wal/instance
I20260807 07:04:38.814417 21836 external_mini_cluster.cc:1428] Setting key d5e2cb0b057d99e975a561300b275074
Aug 07 07:04:38 dist-test-slave-ncp9 krb5kdc[6839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086278, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.1@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:04:38.891199 7786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:38.817527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:45615 (local address 127.21.83.62:38421)
0807 07:04:38.851087 (+ 33560us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:38.851095 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:38.853012 (+ 1917us) server_negotiation.cc:407] Connection header received
0807 07:04:38.857821 (+ 4809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:38.857829 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:38.857960 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:38.858057 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:04:38.868679 (+ 10622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:38.870435 (+ 1756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:38.874977 (+ 4542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:38.876572 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:38.876700 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:38.883238 (+ 6538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:04:38.883246 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:04:38.883251 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:04:38.883324 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:04:38.887087 (+ 3763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:38.888206 (+ 1119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:38.888213 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:38.888216 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:38.888394 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:38.888995 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:38.889000 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:38.889003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:38.889377 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:04:38.889506 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:38.890567 (+ 1061us) server_negotiation.cc:299] Negotiation successful
0807 07:04:38.890836 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33394,"thread_start_us":33234,"threads_started":1}
I20260807 07:04:38.893535 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:38.814573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38421 (local address 127.21.83.1:45615)
0807 07:04:38.851893 (+ 37320us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:38.851983 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0807 07:04:38.857234 (+ 5251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:04:38.858835 (+ 1601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:04:38.858851 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:04:38.862057 (+ 3206us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:04:38.866901 (+ 4844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:38.866953 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:38.870959 (+ 4006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:38.870974 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:38.874730 (+ 3756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:38.874758 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:38.877637 (+ 2879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:38.877654 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:38.878271 (+ 617us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:38.879140 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:04:38.879624 (+ 484us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:04:38.882875 (+ 3251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:04:38.887366 (+ 4491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:04:38.887376 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:04:38.887397 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:04:38.888044 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:04:38.888550 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:04:38.888575 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:04:38.888578 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:04:38.888832 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:04:38.889629 (+ 797us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:04:38.889637 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:04:38.890105 (+ 468us) client_negotiation.cc:769] Sending connection context
0807 07:04:38.891511 (+ 1406us) client_negotiation.cc:240] Negotiation successful
0807 07:04:38.892105 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36837,"thread_start_us":36618,"threads_started":1}
I20260807 07:04:38.895624 7783 heartbeater.cc:347] Connected to a master server at 127.21.83.62:38421
I20260807 07:04:38.895991 7783 heartbeater.cc:464] Registering TS with master...
I20260807 07:04:38.897082 7783 heartbeater.cc:511] Master 127.21.83.62:38421 requested a full tablet report, sending...
I20260807 07:04:38.899986 7484 ts_manager.cc:194] Registered new tserver with Master: e25131831d6a4c2aa8d120b6d65ff5dc (127.21.83.1:45467)
I20260807 07:04:38.902143 7484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.1@KRBTEST.COM'} at 127.21.83.1:45615
WARNING: no policy specified for kudu/127.21.83.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab.
Entry for principal kudu/127.21.83.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab.
W20260807 07:04:38.951643 21836 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.21.83.2: real 0.134s user 0.002s sys 0.028s
I20260807 07:04:38.952348 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.83.20:40945/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.2:0
--local_ip_for_outbound_sockets=127.21.83.2
--webserver_interface=127.21.83.2
--webserver_port=0
--tserver_master_addrs=127.21.83.62:38421
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab
--principal=kudu/127.21.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:43085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 07:04:39 dist-test-slave-ncp9 krb5kdc[6839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086278, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for HTTP/127.21.83.20@KRBTEST.COM
W20260807 07:04:39.471992 7790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:04:39.472334 7790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:04:39.472452 7790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:04:39.480681 7790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:04:39.480831 7790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:04:39.481016 7790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.2
I20260807 07:04:39.494751 7790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:43085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.21.83.20:40945/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab
--principal=kudu/127.21.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.83.2
--webserver_port=0
--tserver_master_addrs=127.21.83.62:38421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.2
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:04:39.496182 7790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:04:39.497635 7790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:04:39.511703 7811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:04:39.511744 7812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:04:39.511997 7815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:04:39.514489 7790 server_base.cc:1034] running on GCE node
I20260807 07:04:39.515249 7790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:04:39.516404 7790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:04:39.517699 7790 hybrid_clock.cc:648] HybridClock initialized: now 1786086279517608 us; error 79 us; skew 500 ppm
I20260807 07:04:39.518177 7790 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:04:39 dist-test-slave-ncp9 krb5kdc[6839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086279, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:04:39.522361 7790 init.cc:377] Logged in from keytab as kudu/127.21.83.2@KRBTEST.COM (short username kudu)
I20260807 07:04:39.524179 7790 webserver.cc:460] Webserver started at http://127.21.83.2:42723/ using document root <none> and password file <none>
I20260807 07:04:39.525040 7790 fs_manager.cc:359] Metadata directory not provided
I20260807 07:04:39.525215 7790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:04:39.525519 7790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 07:04:39 dist-test-slave-ncp9 krb5kdc[6839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086279, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.2@KRBTEST.COM for HTTP/127.21.83.20@KRBTEST.COM
I20260807 07:04:39.546365 7790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-1/data/instance:
uuid: "e2e7caa63ae34259ba99d873914db767"
format_stamp: "Formatted at 2026-08-07 07:04:39 on dist-test-slave-ncp9"
server_key: "e33dd40bb802699bbbe2661f2d560e45"
server_key_iv: "0ac0058439d9d45acf938073e553d45e"
server_key_version: "kuduclusterkey@0"
I20260807 07:04:39.547762 7790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-1/wal/instance:
uuid: "e2e7caa63ae34259ba99d873914db767"
format_stamp: "Formatted at 2026-08-07 07:04:39 on dist-test-slave-ncp9"
server_key: "e33dd40bb802699bbbe2661f2d560e45"
server_key_iv: "0ac0058439d9d45acf938073e553d45e"
server_key_version: "kuduclusterkey@0"
I20260807 07:04:39.555897 7790 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.000s
I20260807 07:04:39.574541 7824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:04:39.576126 7790 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260807 07:04:39.576318 7790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-1/wal
uuid: "e2e7caa63ae34259ba99d873914db767"
format_stamp: "Formatted at 2026-08-07 07:04:39 on dist-test-slave-ncp9"
server_key: "e33dd40bb802699bbbe2661f2d560e45"
server_key_iv: "0ac0058439d9d45acf938073e553d45e"
server_key_version: "kuduclusterkey@0"
I20260807 07:04:39.576542 7790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:04:39.594208 7790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:04:39.598999 7790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:04:39.599293 7790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:04:39.600505 7790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:04:39.602131 7790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:04:39.602231 7790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:04:39.602336 7790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:04:39.602387 7790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:04:39.640635 7790 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.2:46005
I20260807 07:04:39.640690 7969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.2:46005 every 8 connection(s)
I20260807 07:04:39.642132 7790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
I20260807 07:04:39.651264 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 7790
I20260807 07:04:39.651424 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-1/wal/instance
I20260807 07:04:39.651925 21836 external_mini_cluster.cc:1428] Setting key c917fe21922843b191c84c35077c246f
Aug 07 07:04:39 dist-test-slave-ncp9 krb5kdc[6839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086279, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.2@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:04:39.664408 7973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:39.644952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:37527 (local address 127.21.83.62:38421)
0807 07:04:39.645426 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:39.645430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:39.646497 (+ 1067us) server_negotiation.cc:407] Connection header received
0807 07:04:39.648415 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:39.648422 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:39.648513 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:39.648618 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:04:39.650305 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:39.651095 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:39.652888 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:39.653518 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:39.653653 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:39.657696 (+ 4043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:04:39.657702 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:04:39.657705 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:04:39.657748 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:04:39.660645 (+ 2897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:39.661858 (+ 1213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:39.661863 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:39.661866 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:39.662027 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:39.662555 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:39.662558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:39.662560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:39.662873 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:04:39.662990 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:39.663919 (+ 929us) server_negotiation.cc:299] Negotiation successful
0807 07:04:39.664126 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":235,"threads_started":1}
I20260807 07:04:39.665759 7975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:39.645535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38421 (local address 127.21.83.2:37527)
0807 07:04:39.646321 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:39.646362 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 07:04:39.648080 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:04:39.648942 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:04:39.648955 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:04:39.649430 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:04:39.650096 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:39.650116 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:39.651253 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:39.651257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:39.652726 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:39.652742 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:39.653794 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:39.653804 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:39.654114 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:39.654888 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:04:39.654915 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:04:39.657364 (+ 2449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:04:39.660889 (+ 3525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:04:39.660937 (+ 48us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:04:39.660958 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:04:39.661681 (+ 723us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:04:39.662139 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:04:39.662143 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:04:39.662147 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:04:39.662453 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:04:39.662995 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:04:39.663004 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:04:39.663467 (+ 463us) client_negotiation.cc:769] Sending connection context
0807 07:04:39.663782 (+ 315us) client_negotiation.cc:240] Negotiation successful
0807 07:04:39.664348 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":339,"threads_started":1}
I20260807 07:04:39.667573 7971 heartbeater.cc:347] Connected to a master server at 127.21.83.62:38421
I20260807 07:04:39.667896 7971 heartbeater.cc:464] Registering TS with master...
I20260807 07:04:39.668668 7971 heartbeater.cc:511] Master 127.21.83.62:38421 requested a full tablet report, sending...
I20260807 07:04:39.670301 7484 ts_manager.cc:194] Registered new tserver with Master: e2e7caa63ae34259ba99d873914db767 (127.21.83.2:46005)
I20260807 07:04:39.671146 7484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.2@KRBTEST.COM'} at 127.21.83.2:37527
WARNING: no policy specified for kudu/127.21.83.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab.
Entry for principal kudu/127.21.83.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab.
I20260807 07:04:39.725435 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.83.20:40945/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.3:0
--local_ip_for_outbound_sockets=127.21.83.3
--webserver_interface=127.21.83.3
--webserver_port=0
--tserver_master_addrs=127.21.83.62:38421
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab
--principal=kudu/127.21.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:43085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 07:04:39.906527 7783 heartbeater.cc:503] Master 127.21.83.62:38421 was elected leader, sending a full tablet report...
W20260807 07:04:40.103446 7982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:04:40.103785 7982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:04:40.103888 7982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:04:40.111372 7982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:04:40.111505 7982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:04:40.111629 7982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.3
I20260807 07:04:40.121860 7982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:43085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.21.83.20:40945/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab
--principal=kudu/127.21.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.83.3
--webserver_port=0
--tserver_master_addrs=127.21.83.62:38421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.3
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:04:40.123172 7982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:04:40.124461 7982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:04:40.137689 7989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:04:40.138206 7992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:04:40.138463 7990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:04:40.138796 7982 server_base.cc:1034] running on GCE node
I20260807 07:04:40.139457 7982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:04:40.140439 7982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:04:40.141646 7982 hybrid_clock.cc:648] HybridClock initialized: now 1786086280141594 us; error 47 us; skew 500 ppm
I20260807 07:04:40.141978 7982 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:04:40 dist-test-slave-ncp9 krb5kdc[6839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086280, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:04:40.145388 7982 init.cc:377] Logged in from keytab as kudu/127.21.83.3@KRBTEST.COM (short username kudu)
I20260807 07:04:40.147095 7982 webserver.cc:460] Webserver started at http://127.21.83.3:46155/ using document root <none> and password file <none>
I20260807 07:04:40.147845 7982 fs_manager.cc:359] Metadata directory not provided
I20260807 07:04:40.147962 7982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:04:40.148294 7982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 07:04:40 dist-test-slave-ncp9 krb5kdc[6839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086280, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.3@KRBTEST.COM for HTTP/127.21.83.20@KRBTEST.COM
I20260807 07:04:40.166275 7982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-2/data/instance:
uuid: "deacdf33fee447c88139e07de03494e7"
format_stamp: "Formatted at 2026-08-07 07:04:40 on dist-test-slave-ncp9"
server_key: "c92eb4408bfb26dd8524a1c5d2d01665"
server_key_iv: "69fd0803705b9d84a664c83abb204b97"
server_key_version: "kuduclusterkey@0"
I20260807 07:04:40.167378 7982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-2/wal/instance:
uuid: "deacdf33fee447c88139e07de03494e7"
format_stamp: "Formatted at 2026-08-07 07:04:40 on dist-test-slave-ncp9"
server_key: "c92eb4408bfb26dd8524a1c5d2d01665"
server_key_iv: "69fd0803705b9d84a664c83abb204b97"
server_key_version: "kuduclusterkey@0"
I20260807 07:04:40.173398 7982 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260807 07:04:40.189435 8001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:04:40.191951 7982 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260807 07:04:40.192170 7982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-2/wal
uuid: "deacdf33fee447c88139e07de03494e7"
format_stamp: "Formatted at 2026-08-07 07:04:40 on dist-test-slave-ncp9"
server_key: "c92eb4408bfb26dd8524a1c5d2d01665"
server_key_iv: "69fd0803705b9d84a664c83abb204b97"
server_key_version: "kuduclusterkey@0"
I20260807 07:04:40.192425 7982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:04:40.222890 7982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:04:40.227461 7982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:04:40.227804 7982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:04:40.228955 7982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:04:40.230616 7982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:04:40.230706 7982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:04:40.230844 7982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:04:40.230904 7982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:04:40.266542 7982 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.3:42339
I20260807 07:04:40.266594 8159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.3:42339 every 8 connection(s)
I20260807 07:04:40.267933 7982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
I20260807 07:04:40.272215 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 7982
I20260807 07:04:40.272399 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-2/wal/instance
I20260807 07:04:40.273092 21836 external_mini_cluster.cc:1428] Setting key e3049e6aa1d10cf7af0e8beff8fa3c4f
Aug 07 07:04:40 dist-test-slave-ncp9 krb5kdc[6839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086280, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.3@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:04:40.289426 8166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:40.270505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:53385 (local address 127.21.83.62:38421)
0807 07:04:40.271079 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:40.271084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:40.272054 (+ 970us) server_negotiation.cc:407] Connection header received
0807 07:04:40.274152 (+ 2098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:40.274156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:40.274260 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:40.274363 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:04:40.276272 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:40.277054 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:40.278655 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:40.279091 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:40.279203 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:40.282881 (+ 3678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:04:40.282888 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:04:40.282891 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:04:40.282934 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:04:40.285900 (+ 2966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:40.286745 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:40.286751 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:40.286753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:40.286895 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:40.287366 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:40.287370 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:40.287372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:40.287820 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:04:40.287987 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:40.288750 (+ 763us) server_negotiation.cc:299] Negotiation successful
0807 07:04:40.289054 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":268,"threads_started":1}
I20260807 07:04:40.290459 8167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:40.271012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38421 (local address 127.21.83.3:53385)
0807 07:04:40.271852 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:40.271907 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0807 07:04:40.273851 (+ 1944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:04:40.274642 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:04:40.274651 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:04:40.275194 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:04:40.276086 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:40.276124 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:40.277221 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:40.277225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:40.278505 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:40.278518 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:40.279330 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:40.279338 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:40.279572 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:40.280254 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:04:40.280304 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:04:40.282510 (+ 2206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:04:40.286080 (+ 3570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:04:40.286089 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:04:40.286105 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:04:40.286608 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:04:40.287011 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:04:40.287016 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:04:40.287020 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:04:40.287247 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:04:40.287977 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:04:40.287986 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:04:40.288375 (+ 389us) client_negotiation.cc:769] Sending connection context
0807 07:04:40.288643 (+ 268us) client_negotiation.cc:240] Negotiation successful
0807 07:04:40.289243 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":347,"threads_started":1}
I20260807 07:04:40.292109 8162 heartbeater.cc:347] Connected to a master server at 127.21.83.62:38421
I20260807 07:04:40.292430 8162 heartbeater.cc:464] Registering TS with master...
I20260807 07:04:40.293262 8162 heartbeater.cc:511] Master 127.21.83.62:38421 requested a full tablet report, sending...
I20260807 07:04:40.294870 7484 ts_manager.cc:194] Registered new tserver with Master: deacdf33fee447c88139e07de03494e7 (127.21.83.3:42339)
I20260807 07:04:40.295701 7484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.3@KRBTEST.COM'} at 127.21.83.3:53385
I20260807 07:04:40.301283 21836 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 07:04:40.303092 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 7599
Aug 07 07:04:40 dist-test-slave-ncp9 krb5kdc[6839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.1@KRBTEST.COM
I20260807 07:04:40.321250 8170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:40.304185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60448 (local address 127.21.83.1:45467)
0807 07:04:40.305125 (+ 940us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:40.305134 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:40.305175 (+ 41us) server_negotiation.cc:407] Connection header received
0807 07:04:40.305352 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:40.305360 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:40.305629 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:40.305773 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:04:40.307574 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:40.308623 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:40.310325 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:40.310951 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:40.311074 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:40.315641 (+ 4567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:04:40.315659 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:04:40.315675 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:04:40.315721 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:04:40.318597 (+ 2876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:40.319354 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:40.319361 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:40.319363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:40.319513 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:40.319828 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:40.319831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:40.319833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:40.320187 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:04:40.320425 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:40.320677 (+ 252us) server_negotiation.cc:299] Negotiation successful
0807 07:04:40.320932 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":346,"threads_started":1}
I20260807 07:04:40.322714 7775 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:04:40.406934 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 7599
I20260807 07:04:40.430739 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 7790
Aug 07 07:04:40 dist-test-slave-ncp9 krb5kdc[6839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.2@KRBTEST.COM
I20260807 07:04:40.453080 8173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:40.431767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56650 (local address 127.21.83.2:46005)
0807 07:04:40.432366 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:40.432373 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:40.432401 (+ 28us) server_negotiation.cc:407] Connection header received
0807 07:04:40.432467 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:40.432473 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:40.432696 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:40.432835 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:04:40.434237 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:40.435268 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:40.436746 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:40.437451 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:40.437556 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:40.446758 (+ 9202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:04:40.446777 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:04:40.446795 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:04:40.446841 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:04:40.450102 (+ 3261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:40.450925 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:40.450933 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:40.450936 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:40.451101 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:40.451508 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:40.451511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:40.451513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:40.451885 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:04:40.452131 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:40.452427 (+ 296us) server_negotiation.cc:299] Negotiation successful
0807 07:04:40.452659 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":334,"threads_started":1}
I20260807 07:04:40.454677 7964 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:04:40.538146 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 7790
I20260807 07:04:40.557868 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 7982
Aug 07 07:04:40 dist-test-slave-ncp9 krb5kdc[6839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.3@KRBTEST.COM
I20260807 07:04:40.575531 8175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:40.558902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36882 (local address 127.21.83.3:42339)
0807 07:04:40.559594 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:40.559602 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:40.559632 (+ 30us) server_negotiation.cc:407] Connection header received
0807 07:04:40.559706 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:40.559713 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:40.559922 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:40.560058 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:04:40.561465 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:40.562542 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:40.563976 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:40.564597 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:40.564715 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:40.569633 (+ 4918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:04:40.569660 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:04:40.569678 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:04:40.569722 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:04:40.572598 (+ 2876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:40.573476 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:40.573483 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:40.573486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:40.573629 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:40.574008 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:40.574011 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:40.574013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:40.574364 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:04:40.574610 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:40.574945 (+ 335us) server_negotiation.cc:299] Negotiation successful
0807 07:04:40.575164 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":354,"threads_started":1}
I20260807 07:04:40.576975 8155 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:04:40.660882 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 7982
I20260807 07:04:40.680033 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 7437
I20260807 07:04:40.681391 7522 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:04:40.790323 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 7437
I20260807 07:04:40.825045 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:38421
--webserver_interface=127.21.83.62
--webserver_port=36457
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:43085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:38421
--ranger_config_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.83.20:40945/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.83.20:40945/kms with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:04:41.169737 8178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:04:41.170027 8178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:04:41.170158 8178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:04:41.177496 8178 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:04:41.177601 8178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:04:41.177685 8178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:04:41.177757 8178 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:04:41.177821 8178 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:04:41.188405 8178 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:43085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.83.20:40945/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:38421
--ranger_config_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:38421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.83.62
--webserver_port=36457
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:04:41.189744 8178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:04:41.191056 8178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:04:41.204057 8186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:04:41.204089 8189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:04:41.204273 8185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:04:41.205142 8178 server_base.cc:1034] running on GCE node
I20260807 07:04:41.205958 8178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:04:41.207454 8178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:04:41.208667 8178 hybrid_clock.cc:648] HybridClock initialized: now 1786086281208612 us; error 54 us; skew 500 ppm
I20260807 07:04:41.209050 8178 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:04:41 dist-test-slave-ncp9 krb5kdc[6839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086281, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:04:41.212147 8178 init.cc:377] Logged in from keytab as kudu/127.21.83.62@KRBTEST.COM (short username kudu)
I20260807 07:04:41.213660 8178 webserver.cc:460] Webserver started at http://127.21.83.62:36457/ using document root <none> and password file <none>
I20260807 07:04:41.214318 8178 fs_manager.cc:359] Metadata directory not provided
I20260807 07:04:41.214447 8178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 07 07:04:41 dist-test-slave-ncp9 krb5kdc[6839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086281, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for HTTP/127.21.83.20@KRBTEST.COM
I20260807 07:04:41.260955 8178 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260807 07:04:41.264127 8202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:04:41.265563 8178 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260807 07:04:41.265753 8178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "41e179b646cc45b381d9798170401005"
format_stamp: "Formatted at 2026-08-07 07:04:36 on dist-test-slave-ncp9"
server_key: "1fd371de1e352bcde1dddb3b66c7acd9"
server_key_iv: "1eb18f18ccc0f4ace2ca69a790d5ea0d"
server_key_version: "kuduclusterkey@0"
I20260807 07:04:41.266739 8178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:04:41.280493 8178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:04:41.284029 8178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:04:41.284302 8178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:04:41.302371 8178 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.62:38421
I20260807 07:04:41.302400 8271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.62:38421 every 8 connection(s)
I20260807 07:04:41.303896 8178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:04:41.306404 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 8178
I20260807 07:04:41.307243 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.83.20:40945/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.1:45467
--local_ip_for_outbound_sockets=127.21.83.1
--tserver_master_addrs=127.21.83.62:38421
--webserver_port=40549
--webserver_interface=127.21.83.1
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab
--principal=kudu/127.21.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:43085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.83.20:40945/kms with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 07:04:41.310760 8272 sys_catalog.cc:263] Verifying existing consensus state
I20260807 07:04:41.314378 8272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41e179b646cc45b381d9798170401005: Bootstrap starting.
I20260807 07:04:41.333225 8272 log.cc:824] T 00000000000000000000000000000000 P 41e179b646cc45b381d9798170401005: Log is configured to *not* fsync() on all Append() calls
I20260807 07:04:41.342154 8272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41e179b646cc45b381d9798170401005: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 07:04:41.342662 8272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41e179b646cc45b381d9798170401005: Bootstrap complete.
I20260807 07:04:41.351640 8272 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41e179b646cc45b381d9798170401005 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41e179b646cc45b381d9798170401005" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 38421 } }
I20260807 07:04:41.352485 8272 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41e179b646cc45b381d9798170401005 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41e179b646cc45b381d9798170401005, State: Initialized, Role: FOLLOWER
I20260807 07:04:41.353112 8272 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41e179b646cc45b381d9798170401005 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41e179b646cc45b381d9798170401005" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 38421 } }
I20260807 07:04:41.353341 8272 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41e179b646cc45b381d9798170401005 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:04:41.353449 8272 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41e179b646cc45b381d9798170401005 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:04:41.353634 8272 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41e179b646cc45b381d9798170401005 [term 1 FOLLOWER]: Advancing to term 2
I20260807 07:04:41.357399 8272 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41e179b646cc45b381d9798170401005 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41e179b646cc45b381d9798170401005" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 38421 } }
I20260807 07:04:41.357869 8272 leader_election.cc:302] T 00000000000000000000000000000000 P 41e179b646cc45b381d9798170401005 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41e179b646cc45b381d9798170401005; no voters:
I20260807 07:04:41.358409 8272 leader_election.cc:288] T 00000000000000000000000000000000 P 41e179b646cc45b381d9798170401005 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260807 07:04:41.358615 8278 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41e179b646cc45b381d9798170401005 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 07:04:41.359447 8278 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41e179b646cc45b381d9798170401005 [term 2 LEADER]: Becoming Leader. State: Replica: 41e179b646cc45b381d9798170401005, State: Running, Role: LEADER
I20260807 07:04:41.360212 8272 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41e179b646cc45b381d9798170401005 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:04:41.360069 8278 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41e179b646cc45b381d9798170401005 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41e179b646cc45b381d9798170401005" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 38421 } }
I20260807 07:04:41.363081 8283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41e179b646cc45b381d9798170401005 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41e179b646cc45b381d9798170401005. Latest consensus state: current_term: 2 leader_uuid: "41e179b646cc45b381d9798170401005" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41e179b646cc45b381d9798170401005" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 38421 } } }
I20260807 07:04:41.363490 8283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41e179b646cc45b381d9798170401005 [sys.catalog]: This master's current role is: LEADER
I20260807 07:04:41.363353 8279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41e179b646cc45b381d9798170401005 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "41e179b646cc45b381d9798170401005" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41e179b646cc45b381d9798170401005" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 38421 } } }
I20260807 07:04:41.363675 8279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41e179b646cc45b381d9798170401005 [sys.catalog]: This master's current role is: LEADER
I20260807 07:04:41.368683 8292 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:04:41.375098 8292 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:04:41.376480 8292 catalog_manager.cc:1259] Loaded cluster ID: 5db07495b2694d91a7dbc400bc5b80e2
I20260807 07:04:41.376641 8292 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:04:41.378665 8272 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260807 07:04:41.380682 8292 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:04:41.382113 8292 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 41e179b646cc45b381d9798170401005: Loaded TSK: 0
I20260807 07:04:41.383633 8292 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 07:04:41.732491 8274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:04:41.732779 8274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:04:41.732957 8274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:04:41.744208 8274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:04:41.744429 8274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:04:41.744676 8274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.1
I20260807 07:04:41.760484 8274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:43085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.83.20:40945/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab
--principal=kudu/127.21.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.1:45467
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.83.1
--webserver_port=40549
--tserver_master_addrs=127.21.83.62:38421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.1
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:04:41.762600 8274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:04:41.764544 8274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:04:41.795071 8327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:04:41.795769 8329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:04:41.796741 8326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:04:41.796880 8274 server_base.cc:1034] running on GCE node
I20260807 07:04:41.797727 8274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:04:41.799557 8274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:04:41.800834 8274 hybrid_clock.cc:648] HybridClock initialized: now 1786086281800744 us; error 76 us; skew 500 ppm
I20260807 07:04:41.801299 8274 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:04:41 dist-test-slave-ncp9 krb5kdc[6839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086281, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:04:41.806012 8274 init.cc:377] Logged in from keytab as kudu/127.21.83.1@KRBTEST.COM (short username kudu)
I20260807 07:04:41.822105 8274 webserver.cc:460] Webserver started at http://127.21.83.1:40549/ using document root <none> and password file <none>
I20260807 07:04:41.823205 8274 fs_manager.cc:359] Metadata directory not provided
I20260807 07:04:41.823429 8274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 07 07:04:41 dist-test-slave-ncp9 krb5kdc[6839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086281, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.1@KRBTEST.COM for HTTP/127.21.83.20@KRBTEST.COM
I20260807 07:04:41.854935 8274 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.002s sys 0.001s
I20260807 07:04:41.866240 8338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:04:41.869030 8274 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.006s sys 0.004s
I20260807 07:04:41.869292 8274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-0/wal
uuid: "e25131831d6a4c2aa8d120b6d65ff5dc"
format_stamp: "Formatted at 2026-08-07 07:04:38 on dist-test-slave-ncp9"
server_key: "ffc8e1212f57b3c35f8f4b1a210d7a5e"
server_key_iv: "af71529e831f9230a3cb0594c3d6a7d5"
server_key_version: "kuduclusterkey@0"
I20260807 07:04:41.870873 8274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:04:41.921859 8274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:04:41.926101 8274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:04:41.926383 8274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:04:41.927433 8274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:04:41.928834 8274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:04:41.928953 8274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260807 07:04:41.929070 8274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:04:41.929157 8274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:04:41.998301 8274 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.1:45467
I20260807 07:04:41.998350 8496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.1:45467 every 8 connection(s)
I20260807 07:04:41.999787 8274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
I20260807 07:04:42.004338 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 8274
I20260807 07:04:42.005187 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.83.20:40945/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.2:46005
--local_ip_for_outbound_sockets=127.21.83.2
--tserver_master_addrs=127.21.83.62:38421
--webserver_port=42723
--webserver_interface=127.21.83.2
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab
--principal=kudu/127.21.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:43085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.83.20:40945/kms with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 07:04:42 dist-test-slave-ncp9 krb5kdc[6839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086281, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.1@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:04:42.024632 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:42.002997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38421 (local address 127.21.83.1:37541)
0807 07:04:42.003787 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:42.003832 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 07:04:42.005771 (+ 1939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:04:42.007242 (+ 1471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:04:42.007252 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:04:42.007808 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:04:42.008516 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:42.008539 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:42.010090 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:42.010094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:42.011262 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:42.011274 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:42.012236 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:42.012243 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:42.012487 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:42.013750 (+ 1263us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:04:42.013772 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:04:42.015724 (+ 1952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:04:42.019565 (+ 3841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:04:42.019574 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:04:42.019587 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:04:42.020142 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:04:42.020643 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:04:42.020647 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:04:42.020650 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:04:42.020958 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:04:42.021901 (+ 943us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:04:42.021908 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:04:42.022283 (+ 375us) client_negotiation.cc:769] Sending connection context
0807 07:04:42.022756 (+ 473us) client_negotiation.cc:240] Negotiation successful
0807 07:04:42.023325 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":325,"threads_started":1}
I20260807 07:04:42.027870 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:42.003036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:37541 (local address 127.21.83.62:38421)
0807 07:04:42.004540 (+ 1504us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:42.004557 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:42.004604 (+ 47us) server_negotiation.cc:407] Connection header received
0807 07:04:42.006293 (+ 1689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:42.006305 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:42.006600 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:42.006932 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:04:42.008714 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:42.009921 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:42.011400 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:42.012018 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:42.012187 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:42.015946 (+ 3759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:04:42.015953 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:04:42.015965 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:04:42.016012 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:04:42.019399 (+ 3387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:42.020320 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:42.020328 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:42.020335 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:42.020519 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:42.021132 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:42.021136 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:42.021139 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:42.021744 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:04:42.025928 (+ 4184us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:42.026226 (+ 298us) server_negotiation.cc:299] Negotiation successful
0807 07:04:42.026702 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1262,"thread_start_us":1070,"threads_started":1}
I20260807 07:04:42.031723 8500 heartbeater.cc:347] Connected to a master server at 127.21.83.62:38421
I20260807 07:04:42.032078 8500 heartbeater.cc:464] Registering TS with master...
I20260807 07:04:42.033046 8500 heartbeater.cc:511] Master 127.21.83.62:38421 requested a full tablet report, sending...
W20260807 07:04:42.036505 8500 heartbeater.cc:644] Failed to heartbeat to 127.21.83.62:38421 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260807 07:04:42.036767 8500 heartbeater.cc:464] Registering TS with master...
I20260807 07:04:42.037112 8500 heartbeater.cc:511] Master 127.21.83.62:38421 requested a full tablet report, sending...
I20260807 07:04:42.038678 8500 heartbeater.cc:464] Registering TS with master...
I20260807 07:04:42.039142 8500 heartbeater.cc:511] Master 127.21.83.62:38421 requested a full tablet report, sending...
W20260807 07:04:42.040436 8500 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 07:04:42.398892 8504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:04:42.399164 8504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:04:42.399277 8504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:04:42.406847 8504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:04:42.406987 8504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:04:42.407187 8504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.2
I20260807 07:04:42.422854 8504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:43085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.83.20:40945/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab
--principal=kudu/127.21.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.2:46005
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.83.2
--webserver_port=42723
--tserver_master_addrs=127.21.83.62:38421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.2
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:04:42.424911 8504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:04:42.426769 8504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:04:42.443845 8523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:04:42.446173 8524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:04:42.446825 8527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:04:42.447856 8504 server_base.cc:1034] running on GCE node
I20260807 07:04:42.448617 8504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:04:42.449832 8504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:04:42.451077 8504 hybrid_clock.cc:648] HybridClock initialized: now 1786086282451001 us; error 57 us; skew 500 ppm
I20260807 07:04:42.451434 8504 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:04:42 dist-test-slave-ncp9 krb5kdc[6839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086282, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:04:42.455047 8504 init.cc:377] Logged in from keytab as kudu/127.21.83.2@KRBTEST.COM (short username kudu)
I20260807 07:04:42.456969 8504 webserver.cc:460] Webserver started at http://127.21.83.2:42723/ using document root <none> and password file <none>
I20260807 07:04:42.457885 8504 fs_manager.cc:359] Metadata directory not provided
I20260807 07:04:42.458051 8504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 07 07:04:42 dist-test-slave-ncp9 krb5kdc[6839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086282, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.2@KRBTEST.COM for HTTP/127.21.83.20@KRBTEST.COM
I20260807 07:04:42.486562 8504 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.003s sys 0.005s
I20260807 07:04:42.496323 8538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:04:42.498296 8504 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.007s sys 0.001s
I20260807 07:04:42.498594 8504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-1/wal
uuid: "e2e7caa63ae34259ba99d873914db767"
format_stamp: "Formatted at 2026-08-07 07:04:39 on dist-test-slave-ncp9"
server_key: "e33dd40bb802699bbbe2661f2d560e45"
server_key_iv: "0ac0058439d9d45acf938073e553d45e"
server_key_version: "kuduclusterkey@0"
I20260807 07:04:42.500142 8504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:04:42.561249 8504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:04:42.571414 8504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:04:42.571883 8504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:04:42.574941 8504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:04:42.591177 8504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:04:42.591419 8504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260807 07:04:42.591627 8504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:04:42.591758 8504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 07:04:42 dist-test-slave-ncp9 krb5kdc[6839](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786086282, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:04:42.772131 8504 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.2:46005
I20260807 07:04:42.772182 8683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.2:46005 every 8 connection(s)
I20260807 07:04:42.774278 8504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
I20260807 07:04:42.781536 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 8504
I20260807 07:04:42.782389 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.83.20:40945/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.3:42339
--local_ip_for_outbound_sockets=127.21.83.3
--tserver_master_addrs=127.21.83.62:38421
--webserver_port=46155
--webserver_interface=127.21.83.3
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab
--principal=kudu/127.21.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:43085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.83.20:40945/kms with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 07:04:42 dist-test-slave-ncp9 krb5kdc[6839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086282, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.2@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:04:42.826594 8688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:42.778326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38421 (local address 127.21.83.2:50989)
0807 07:04:42.794051 (+ 15725us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:42.794119 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0807 07:04:42.796541 (+ 2422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:04:42.802541 (+ 6000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:04:42.802553 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:04:42.803180 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:04:42.803990 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:42.804015 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:42.806134 (+ 2119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:42.806139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:42.808416 (+ 2277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:42.808438 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:42.809906 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:42.809913 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:42.810238 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:42.811118 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:04:42.811152 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:04:42.813867 (+ 2715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:04:42.819909 (+ 6042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:04:42.819922 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:04:42.819947 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:04:42.820721 (+ 774us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:04:42.821331 (+ 610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:04:42.821337 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:04:42.821341 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:04:42.821937 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:04:42.822856 (+ 919us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:04:42.822866 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:04:42.824032 (+ 1166us) client_negotiation.cc:769] Sending connection context
0807 07:04:42.824426 (+ 394us) client_negotiation.cc:240] Negotiation successful
0807 07:04:42.825119 (+ 693us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15390,"thread_start_us":15175,"threads_started":1}
I20260807 07:04:42.832582 8687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:42.777681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:50989 (local address 127.21.83.62:38421)
0807 07:04:42.801857 (+ 24176us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:42.801864 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:42.801887 (+ 23us) server_negotiation.cc:407] Connection header received
0807 07:04:42.801959 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:42.801965 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:42.802058 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:42.802214 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:04:42.804721 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:42.805911 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:42.808729 (+ 2818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:42.809413 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:42.809543 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:42.814226 (+ 4683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:04:42.814234 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:04:42.814238 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:04:42.814286 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:04:42.819628 (+ 5342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:42.820971 (+ 1343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:42.820976 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:42.820980 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:42.821169 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:42.822137 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:42.822143 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:42.822146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:42.822593 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:04:42.822746 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:42.831905 (+ 9159us) server_negotiation.cc:299] Negotiation successful
0807 07:04:42.832199 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24017,"thread_start_us":24657,"threads_started":1}
I20260807 07:04:42.837214 8684 heartbeater.cc:347] Connected to a master server at 127.21.83.62:38421
I20260807 07:04:42.837587 8684 heartbeater.cc:464] Registering TS with master...
I20260807 07:04:42.838474 8684 heartbeater.cc:511] Master 127.21.83.62:38421 requested a full tablet report, sending...
I20260807 07:04:42.840658 8226 ts_manager.cc:194] Registered new tserver with Master: e2e7caa63ae34259ba99d873914db767 (127.21.83.2:46005)
I20260807 07:04:42.842907 8226 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.2@KRBTEST.COM'} at 127.21.83.2:50989
I20260807 07:04:43.040604 8500 heartbeater.cc:464] Registering TS with master...
I20260807 07:04:43.041126 8500 heartbeater.cc:511] Master 127.21.83.62:38421 requested a full tablet report, sending...
I20260807 07:04:43.042611 8226 ts_manager.cc:194] Registered new tserver with Master: e25131831d6a4c2aa8d120b6d65ff5dc (127.21.83.1:45467)
I20260807 07:04:43.043766 8226 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.1@KRBTEST.COM'} at 127.21.83.1:37541
Aug 07 07:04:43 dist-test-slave-ncp9 krb5kdc[6839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086282, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for HTTP/127.21.83.20@KRBTEST.COM
W20260807 07:04:43.285640 8690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:04:43.286036 8690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:04:43.286197 8690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:04:43.297499 8690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:04:43.297648 8690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:04:43.297834 8690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.3
I20260807 07:04:43.310411 8690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:43085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.83.20:40945/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab
--principal=kudu/127.21.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.3:42339
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.83.3
--webserver_port=46155
--tserver_master_addrs=127.21.83.62:38421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.3
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:04:43.311710 8690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:04:43.312937 8690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:04:43.326684 8695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:04:43.326655 8696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:04:43.327369 8700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:04:43.327713 8690 server_base.cc:1034] running on GCE node
I20260807 07:04:43.328383 8690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:04:43.329526 8690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:04:43.330790 8690 hybrid_clock.cc:648] HybridClock initialized: now 1786086283330690 us; error 72 us; skew 500 ppm
I20260807 07:04:43.331164 8690 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:04:43 dist-test-slave-ncp9 krb5kdc[6839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086283, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:04:43.335275 8690 init.cc:377] Logged in from keytab as kudu/127.21.83.3@KRBTEST.COM (short username kudu)
I20260807 07:04:43.337190 8690 webserver.cc:460] Webserver started at http://127.21.83.3:46155/ using document root <none> and password file <none>
I20260807 07:04:43.338106 8690 fs_manager.cc:359] Metadata directory not provided
I20260807 07:04:43.338286 8690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 07 07:04:43 dist-test-slave-ncp9 krb5kdc[6839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086283, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.3@KRBTEST.COM for HTTP/127.21.83.20@KRBTEST.COM
I20260807 07:04:43.370728 8690 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260807 07:04:43.374293 8710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:04:43.375945 8690 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 07:04:43.376122 8690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-2/wal
uuid: "deacdf33fee447c88139e07de03494e7"
format_stamp: "Formatted at 2026-08-07 07:04:40 on dist-test-slave-ncp9"
server_key: "c92eb4408bfb26dd8524a1c5d2d01665"
server_key_iv: "69fd0803705b9d84a664c83abb204b97"
server_key_version: "kuduclusterkey@0"
I20260807 07:04:43.377487 8690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:04:43.397811 8690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:04:43.403282 8690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:04:43.403641 8690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:04:43.405015 8690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:04:43.406953 8690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:04:43.407074 8690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:04:43.407251 8690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:04:43.407343 8690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:04:43.468799 8690 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.3:42339
I20260807 07:04:43.468869 8859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.3:42339 every 8 connection(s)
I20260807 07:04:43.471143 8690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
I20260807 07:04:43.481529 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 8690
Aug 07 07:04:43 dist-test-slave-ncp9 krb5kdc[6839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086283, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.3@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:04:43.543905 8865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:43.475919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:38421 (local address 127.21.83.3:39387)
0807 07:04:43.513443 (+ 37524us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:43.513510 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0807 07:04:43.516202 (+ 2692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:04:43.522717 (+ 6515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:04:43.522729 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:04:43.523425 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:04:43.524255 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:43.524290 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:43.526577 (+ 2287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:43.526583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:43.528524 (+ 1941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:43.528539 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:43.529752 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:43.529760 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:43.530041 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:43.530847 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:04:43.530877 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:04:43.533512 (+ 2635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:04:43.538339 (+ 4827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:04:43.538348 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:04:43.538363 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:04:43.538930 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:04:43.539396 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:04:43.539401 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:04:43.539405 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:04:43.539664 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:04:43.540381 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:04:43.540389 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:04:43.540754 (+ 365us) client_negotiation.cc:769] Sending connection context
0807 07:04:43.541542 (+ 788us) client_negotiation.cc:240] Negotiation successful
0807 07:04:43.542055 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37115,"thread_start_us":41129,"threads_started":1}
I20260807 07:04:43.544325 8867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:43.476570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:39387 (local address 127.21.83.62:38421)
0807 07:04:43.511866 (+ 35296us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:43.511873 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:43.516665 (+ 4792us) server_negotiation.cc:407] Connection header received
0807 07:04:43.516744 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:43.516750 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:43.516851 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:43.516998 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:04:43.525212 (+ 8214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:43.526369 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:43.528692 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:43.529494 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:43.529597 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:43.533747 (+ 4150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:04:43.533754 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:04:43.533759 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:04:43.533815 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:04:43.538171 (+ 4356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:43.539079 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:43.539097 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:43.539101 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:43.539252 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:43.539803 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:43.539809 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:43.539813 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:43.540149 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:04:43.540287 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:43.542541 (+ 2254us) server_negotiation.cc:299] Negotiation successful
0807 07:04:43.543928 (+ 1387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":35136,"thread_start_us":35313,"threads_started":1}
I20260807 07:04:43.545905 8860 heartbeater.cc:347] Connected to a master server at 127.21.83.62:38421
I20260807 07:04:43.546253 8860 heartbeater.cc:464] Registering TS with master...
I20260807 07:04:43.547091 8860 heartbeater.cc:511] Master 127.21.83.62:38421 requested a full tablet report, sending...
I20260807 07:04:43.548658 8226 ts_manager.cc:194] Registered new tserver with Master: deacdf33fee447c88139e07de03494e7 (127.21.83.3:42339)
I20260807 07:04:43.549875 8226 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.3@KRBTEST.COM'} at 127.21.83.3:39387
I20260807 07:04:43.553428 8869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:43.512081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53236 (local address 127.21.83.62:38421)
0807 07:04:43.518299 (+ 6218us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:43.518305 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:43.518323 (+ 18us) server_negotiation.cc:407] Connection header received
0807 07:04:43.518372 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:43.518377 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:43.518466 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:43.520944 (+ 2478us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:04:43.521175 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:43.522276 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:43.528752 (+ 6476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:43.536304 (+ 7552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:43.536428 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:43.540236 (+ 3808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:04:43.540244 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:04:43.540248 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:04:43.540290 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:04:43.543757 (+ 3467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:43.548433 (+ 4676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:43.548438 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:43.548441 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:43.548577 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:43.549060 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:43.549065 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:43.549068 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:43.549437 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:04:43.549711 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:43.552790 (+ 3079us) server_negotiation.cc:299] Negotiation successful
0807 07:04:43.553046 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":6102,"thread_start_us":6484,"threads_started":1}
I20260807 07:04:43.556263 21836 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 07:04:43 dist-test-slave-ncp9 krb5kdc[6839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086282, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for HTTP/127.21.83.20@KRBTEST.COM
I20260807 07:04:43.584729 8869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:43.560742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53248 (local address 127.21.83.62:38421)
0807 07:04:43.560990 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:43.560996 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:43.567512 (+ 6516us) server_negotiation.cc:407] Connection header received
0807 07:04:43.567680 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:43.567686 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:43.567783 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:43.567888 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:04:43.569881 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:43.570584 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:43.572556 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:43.573165 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:43.573272 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:43.576592 (+ 3320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:04:43.576600 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:04:43.576605 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:04:43.576647 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:04:43.581269 (+ 4622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:43.582183 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:43.582203 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:43.582207 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:43.582353 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:43.583057 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:43.583062 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:43.583066 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:43.583455 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:04:43.583675 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:43.584187 (+ 512us) server_negotiation.cc:299] Negotiation successful
0807 07:04:43.584409 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260807 07:04:43.589694 8226 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53248:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 07:04:43.598950 8226 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 07:04:43.652205 8886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:43.620980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:45467 (local address 127.0.0.1:60460)
0807 07:04:43.632112 (+ 11132us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:43.632186 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0807 07:04:43.632401 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:04:43.634625 (+ 2224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:04:43.634634 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:04:43.634671 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:04:43.635104 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:43.635117 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:43.637185 (+ 2068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:43.637191 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:43.639510 (+ 2319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:43.639526 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:43.651183 (+ 11657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:43.651191 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:43.651429 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:43.651481 (+ 52us) client_negotiation.cc:769] Sending connection context
0807 07:04:43.651620 (+ 139us) client_negotiation.cc:240] Negotiation successful
0807 07:04:43.651828 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10804,"thread_start_us":18762,"threads_started":1}
I20260807 07:04:43.654189 8885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:43.621642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60460 (local address 127.21.83.1:45467)
0807 07:04:43.633696 (+ 12054us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:43.633706 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:43.633740 (+ 34us) server_negotiation.cc:407] Connection header received
0807 07:04:43.633842 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:43.633849 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:43.634117 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:43.634338 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:04:43.635347 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:43.636998 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:43.647728 (+ 10730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:43.650607 (+ 2879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:43.650750 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:43.651013 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:43.653596 (+ 2583us) server_negotiation.cc:299] Negotiation successful
0807 07:04:43.653804 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11865,"thread_start_us":11693,"threads_started":1}
I20260807 07:04:43.659328 8405 tablet_service.cc:1467] Processing CreateTablet for tablet 5b4b2bb509514cbd9b1f57197e6e2dc7 (DEFAULT_TABLE table=test-table [id=6594e40bad5b464e858434a4ff2a219f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:04:43.661166 8405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b4b2bb509514cbd9b1f57197e6e2dc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:04:43.664253 8886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:43.640199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.2:46005 (local address 127.0.0.1:56652)
0807 07:04:43.652455 (+ 12256us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:43.652474 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 07:04:43.652565 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:04:43.657949 (+ 5384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:04:43.657954 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:04:43.657980 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:04:43.658142 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:43.658164 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:43.660007 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:43.660012 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:43.661695 (+ 1683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:43.661719 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:43.663495 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:43.663502 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:43.663678 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:43.663722 (+ 44us) client_negotiation.cc:769] Sending connection context
0807 07:04:43.663791 (+ 69us) client_negotiation.cc:240] Negotiation successful
0807 07:04:43.663940 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12164,"thread_start_us":14578,"threads_started":1}
I20260807 07:04:43.664868 8882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:43.621708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.3:42339 (local address 127.0.0.1:36888)
0807 07:04:43.640012 (+ 18304us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:43.640031 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 07:04:43.646346 (+ 6315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:04:43.656393 (+ 10047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:04:43.656398 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:04:43.656427 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:04:43.656801 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:43.656815 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:43.658929 (+ 2114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:43.658934 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:43.662285 (+ 3351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:43.662300 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:43.664147 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:43.664154 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:43.664319 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:43.664351 (+ 32us) client_negotiation.cc:769] Sending connection context
0807 07:04:43.664421 (+ 70us) client_negotiation.cc:240] Negotiation successful
0807 07:04:43.664548 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18178,"thread_start_us":10906,"threads_started":1}
I20260807 07:04:43.665496 8890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:43.640427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56652 (local address 127.21.83.2:46005)
0807 07:04:43.657201 (+ 16774us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:43.657211 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:43.657238 (+ 27us) server_negotiation.cc:407] Connection header received
0807 07:04:43.657331 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:43.657339 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:43.657552 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:43.657735 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:04:43.658409 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:43.659844 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:43.661906 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:43.663312 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:43.664612 (+ 1300us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:43.664815 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:43.664886 (+ 71us) server_negotiation.cc:299] Negotiation successful
0807 07:04:43.665215 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16595,"thread_start_us":15629,"threads_started":1}
I20260807 07:04:43.665892 8887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:43.622390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36888 (local address 127.21.83.3:42339)
0807 07:04:43.655688 (+ 33298us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:43.655696 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:43.655729 (+ 33us) server_negotiation.cc:407] Connection header received
0807 07:04:43.655834 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:43.655842 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:43.656163 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:43.656418 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:04:43.657082 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:43.658764 (+ 1682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:43.662486 (+ 3722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:43.663921 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:43.665099 (+ 1178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:43.665318 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:43.665386 (+ 68us) server_negotiation.cc:299] Negotiation successful
0807 07:04:43.665545 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33107,"thread_start_us":34283,"threads_started":1}
I20260807 07:04:43.669226 8774 tablet_service.cc:1467] Processing CreateTablet for tablet 5b4b2bb509514cbd9b1f57197e6e2dc7 (DEFAULT_TABLE table=test-table [id=6594e40bad5b464e858434a4ff2a219f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:04:43.669226 8603 tablet_service.cc:1467] Processing CreateTablet for tablet 5b4b2bb509514cbd9b1f57197e6e2dc7 (DEFAULT_TABLE table=test-table [id=6594e40bad5b464e858434a4ff2a219f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:04:43.670715 8603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b4b2bb509514cbd9b1f57197e6e2dc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:04:43.670715 8774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b4b2bb509514cbd9b1f57197e6e2dc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:04:43.689728 8891 tablet_bootstrap.cc:492] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P e25131831d6a4c2aa8d120b6d65ff5dc: Bootstrap starting.
I20260807 07:04:43.694185 8891 tablet_bootstrap.cc:654] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P e25131831d6a4c2aa8d120b6d65ff5dc: Neither blocks nor log segments found. Creating new log.
I20260807 07:04:43.698524 8894 tablet_bootstrap.cc:492] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P deacdf33fee447c88139e07de03494e7: Bootstrap starting.
I20260807 07:04:43.707214 8894 tablet_bootstrap.cc:654] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P deacdf33fee447c88139e07de03494e7: Neither blocks nor log segments found. Creating new log.
I20260807 07:04:43.708848 8894 log.cc:824] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P deacdf33fee447c88139e07de03494e7: Log is configured to *not* fsync() on all Append() calls
I20260807 07:04:43.710417 8892 tablet_bootstrap.cc:492] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P e2e7caa63ae34259ba99d873914db767: Bootstrap starting.
I20260807 07:04:43.712987 8891 log.cc:824] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P e25131831d6a4c2aa8d120b6d65ff5dc: Log is configured to *not* fsync() on all Append() calls
I20260807 07:04:43.714636 8892 tablet_bootstrap.cc:654] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P e2e7caa63ae34259ba99d873914db767: Neither blocks nor log segments found. Creating new log.
I20260807 07:04:43.715755 8892 log.cc:824] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P e2e7caa63ae34259ba99d873914db767: Log is configured to *not* fsync() on all Append() calls
I20260807 07:04:43.719136 8891 tablet_bootstrap.cc:492] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P e25131831d6a4c2aa8d120b6d65ff5dc: No bootstrap required, opened a new log
I20260807 07:04:43.719528 8891 ts_tablet_manager.cc:1397] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P e25131831d6a4c2aa8d120b6d65ff5dc: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.004s
I20260807 07:04:43.727372 8892 tablet_bootstrap.cc:492] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P e2e7caa63ae34259ba99d873914db767: No bootstrap required, opened a new log
I20260807 07:04:43.727725 8892 ts_tablet_manager.cc:1397] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P e2e7caa63ae34259ba99d873914db767: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.006s
I20260807 07:04:43.731180 8891 raft_consensus.cc:348] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P e25131831d6a4c2aa8d120b6d65ff5dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e25131831d6a4c2aa8d120b6d65ff5dc" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 45467 } } peers { permanent_uuid: "e2e7caa63ae34259ba99d873914db767" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 46005 } } peers { permanent_uuid: "deacdf33fee447c88139e07de03494e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 42339 } }
I20260807 07:04:43.731698 8891 raft_consensus.cc:374] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P e25131831d6a4c2aa8d120b6d65ff5dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:04:43.731814 8891 raft_consensus.cc:729] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P e25131831d6a4c2aa8d120b6d65ff5dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e25131831d6a4c2aa8d120b6d65ff5dc, State: Initialized, Role: FOLLOWER
I20260807 07:04:43.732489 8891 consensus_queue.cc:260] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P e25131831d6a4c2aa8d120b6d65ff5dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e25131831d6a4c2aa8d120b6d65ff5dc" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 45467 } } peers { permanent_uuid: "e2e7caa63ae34259ba99d873914db767" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 46005 } } peers { permanent_uuid: "deacdf33fee447c88139e07de03494e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 42339 } }
I20260807 07:04:43.740283 8894 tablet_bootstrap.cc:492] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P deacdf33fee447c88139e07de03494e7: No bootstrap required, opened a new log
I20260807 07:04:43.740661 8894 ts_tablet_manager.cc:1397] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P deacdf33fee447c88139e07de03494e7: Time spent bootstrapping tablet: real 0.043s user 0.008s sys 0.000s
I20260807 07:04:43.742388 8500 heartbeater.cc:503] Master 127.21.83.62:38421 was elected leader, sending a full tablet report...
I20260807 07:04:43.743039 8891 ts_tablet_manager.cc:1428] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P e25131831d6a4c2aa8d120b6d65ff5dc: Time spent starting tablet: real 0.023s user 0.021s sys 0.003s
I20260807 07:04:43.745128 8892 raft_consensus.cc:348] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P e2e7caa63ae34259ba99d873914db767 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e25131831d6a4c2aa8d120b6d65ff5dc" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 45467 } } peers { permanent_uuid: "e2e7caa63ae34259ba99d873914db767" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 46005 } } peers { permanent_uuid: "deacdf33fee447c88139e07de03494e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 42339 } }
I20260807 07:04:43.745617 8892 raft_consensus.cc:374] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P e2e7caa63ae34259ba99d873914db767 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:04:43.745736 8892 raft_consensus.cc:729] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P e2e7caa63ae34259ba99d873914db767 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2e7caa63ae34259ba99d873914db767, State: Initialized, Role: FOLLOWER
I20260807 07:04:43.746462 8892 consensus_queue.cc:260] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P e2e7caa63ae34259ba99d873914db767 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e25131831d6a4c2aa8d120b6d65ff5dc" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 45467 } } peers { permanent_uuid: "e2e7caa63ae34259ba99d873914db767" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 46005 } } peers { permanent_uuid: "deacdf33fee447c88139e07de03494e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 42339 } }
W20260807 07:04:43.753417 8501 tablet.cc:2369] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P e25131831d6a4c2aa8d120b6d65ff5dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 07:04:43.761440 8894 raft_consensus.cc:348] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P deacdf33fee447c88139e07de03494e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e25131831d6a4c2aa8d120b6d65ff5dc" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 45467 } } peers { permanent_uuid: "e2e7caa63ae34259ba99d873914db767" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 46005 } } peers { permanent_uuid: "deacdf33fee447c88139e07de03494e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 42339 } }
I20260807 07:04:43.762072 8894 raft_consensus.cc:374] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P deacdf33fee447c88139e07de03494e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:04:43.762213 8894 raft_consensus.cc:729] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P deacdf33fee447c88139e07de03494e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: deacdf33fee447c88139e07de03494e7, State: Initialized, Role: FOLLOWER
I20260807 07:04:43.763372 8684 heartbeater.cc:503] Master 127.21.83.62:38421 was elected leader, sending a full tablet report...
I20260807 07:04:43.764195 8892 ts_tablet_manager.cc:1428] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P e2e7caa63ae34259ba99d873914db767: Time spent starting tablet: real 0.036s user 0.019s sys 0.011s
I20260807 07:04:43.765017 8894 consensus_queue.cc:260] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P deacdf33fee447c88139e07de03494e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e25131831d6a4c2aa8d120b6d65ff5dc" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 45467 } } peers { permanent_uuid: "e2e7caa63ae34259ba99d873914db767" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 46005 } } peers { permanent_uuid: "deacdf33fee447c88139e07de03494e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 42339 } }
I20260807 07:04:43.780359 8860 heartbeater.cc:503] Master 127.21.83.62:38421 was elected leader, sending a full tablet report...
I20260807 07:04:43.782984 8894 ts_tablet_manager.cc:1428] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P deacdf33fee447c88139e07de03494e7: Time spent starting tablet: real 0.042s user 0.015s sys 0.004s
W20260807 07:04:43.791663 8685 tablet.cc:2369] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P e2e7caa63ae34259ba99d873914db767: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 07:04:43.917120 8903 raft_consensus.cc:482] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P deacdf33fee447c88139e07de03494e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:04:43.917443 8903 raft_consensus.cc:504] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P deacdf33fee447c88139e07de03494e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e25131831d6a4c2aa8d120b6d65ff5dc" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 45467 } } peers { permanent_uuid: "e2e7caa63ae34259ba99d873914db767" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 46005 } } peers { permanent_uuid: "deacdf33fee447c88139e07de03494e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 42339 } }
I20260807 07:04:43.919186 8903 leader_election.cc:288] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P deacdf33fee447c88139e07de03494e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e25131831d6a4c2aa8d120b6d65ff5dc (127.21.83.1:45467), e2e7caa63ae34259ba99d873914db767 (127.21.83.2:46005)
I20260807 07:04:43.925841 8885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:43.919796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:39615 (local address 127.21.83.1:45467)
0807 07:04:43.920016 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:43.920022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:43.920044 (+ 22us) server_negotiation.cc:407] Connection header received
0807 07:04:43.920191 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:43.920195 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:43.920255 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:43.920329 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:04:43.920878 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:43.921887 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:43.924129 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:43.924934 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:43.925077 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:43.925188 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:43.925457 (+ 269us) server_negotiation.cc:299] Negotiation successful
0807 07:04:43.925591 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 07:04:43.925841 8865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:43.919672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:45467 (local address 127.21.83.3:39615)
0807 07:04:43.919889 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:43.919911 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 07:04:43.920054 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:04:43.920379 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:04:43.920385 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:04:43.920412 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:04:43.920682 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:43.920693 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:43.922059 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:43.922063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:43.923973 (+ 1910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:43.923986 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:43.925116 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:43.925122 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:43.925305 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:43.925330 (+ 25us) client_negotiation.cc:769] Sending connection context
0807 07:04:43.925416 (+ 86us) client_negotiation.cc:240] Negotiation successful
0807 07:04:43.925571 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260807 07:04:43.926894 8913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:43.920120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.2:46005 (local address 127.21.83.3:48135)
0807 07:04:43.921098 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:43.921116 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 07:04:43.921219 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:04:43.921580 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:04:43.921584 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:04:43.921609 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:04:43.921965 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:43.921978 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:43.923416 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:43.923420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:43.924836 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:43.924850 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:43.926223 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:43.926229 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:43.926399 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:43.926431 (+ 32us) client_negotiation.cc:769] Sending connection context
0807 07:04:43.926522 (+ 91us) client_negotiation.cc:240] Negotiation successful
0807 07:04:43.926634 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":855,"thread_start_us":660,"threads_started":1}
I20260807 07:04:43.926939 8890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:43.920736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:48135 (local address 127.21.83.2:46005)
0807 07:04:43.921153 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:43.921160 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:43.921181 (+ 21us) server_negotiation.cc:407] Connection header received
0807 07:04:43.921374 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:43.921379 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:43.921457 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:43.921541 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:04:43.922179 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:43.923251 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:43.925000 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:43.925849 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:43.925973 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:43.926074 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:43.926558 (+ 484us) server_negotiation.cc:299] Negotiation successful
0807 07:04:43.926702 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267}
I20260807 07:04:43.927000 8429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b4b2bb509514cbd9b1f57197e6e2dc7" candidate_uuid: "deacdf33fee447c88139e07de03494e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e25131831d6a4c2aa8d120b6d65ff5dc" is_pre_election: true
I20260807 07:04:43.927584 8429 raft_consensus.cc:2454] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P e25131831d6a4c2aa8d120b6d65ff5dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate deacdf33fee447c88139e07de03494e7 in term 0.
I20260807 07:04:43.928030 8627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b4b2bb509514cbd9b1f57197e6e2dc7" candidate_uuid: "deacdf33fee447c88139e07de03494e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2e7caa63ae34259ba99d873914db767" is_pre_election: true
I20260807 07:04:43.928297 8715 leader_election.cc:302] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P deacdf33fee447c88139e07de03494e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: deacdf33fee447c88139e07de03494e7, e25131831d6a4c2aa8d120b6d65ff5dc; no voters:
I20260807 07:04:43.928602 8627 raft_consensus.cc:2454] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P e2e7caa63ae34259ba99d873914db767 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate deacdf33fee447c88139e07de03494e7 in term 0.
I20260807 07:04:43.928767 8903 raft_consensus.cc:2781] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P deacdf33fee447c88139e07de03494e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 07:04:43.928962 8903 raft_consensus.cc:482] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P deacdf33fee447c88139e07de03494e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:04:43.929078 8903 raft_consensus.cc:3037] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P deacdf33fee447c88139e07de03494e7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:04:43.931357 8903 raft_consensus.cc:504] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P deacdf33fee447c88139e07de03494e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e25131831d6a4c2aa8d120b6d65ff5dc" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 45467 } } peers { permanent_uuid: "e2e7caa63ae34259ba99d873914db767" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 46005 } } peers { permanent_uuid: "deacdf33fee447c88139e07de03494e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 42339 } }
I20260807 07:04:43.931964 8903 leader_election.cc:288] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P deacdf33fee447c88139e07de03494e7 [CANDIDATE]: Term 1 election: Requested vote from peers e25131831d6a4c2aa8d120b6d65ff5dc (127.21.83.1:45467), e2e7caa63ae34259ba99d873914db767 (127.21.83.2:46005)
I20260807 07:04:43.932535 8429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b4b2bb509514cbd9b1f57197e6e2dc7" candidate_uuid: "deacdf33fee447c88139e07de03494e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e25131831d6a4c2aa8d120b6d65ff5dc"
I20260807 07:04:43.932578 8627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b4b2bb509514cbd9b1f57197e6e2dc7" candidate_uuid: "deacdf33fee447c88139e07de03494e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2e7caa63ae34259ba99d873914db767"
I20260807 07:04:43.932752 8429 raft_consensus.cc:3037] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P e25131831d6a4c2aa8d120b6d65ff5dc [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:04:43.932777 8627 raft_consensus.cc:3037] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P e2e7caa63ae34259ba99d873914db767 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:04:43.935014 8429 raft_consensus.cc:2454] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P e25131831d6a4c2aa8d120b6d65ff5dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate deacdf33fee447c88139e07de03494e7 in term 1.
I20260807 07:04:43.935010 8627 raft_consensus.cc:2454] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P e2e7caa63ae34259ba99d873914db767 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate deacdf33fee447c88139e07de03494e7 in term 1.
I20260807 07:04:43.935436 8715 leader_election.cc:302] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P deacdf33fee447c88139e07de03494e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: deacdf33fee447c88139e07de03494e7, e25131831d6a4c2aa8d120b6d65ff5dc; no voters:
I20260807 07:04:43.935750 8903 raft_consensus.cc:2781] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P deacdf33fee447c88139e07de03494e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:04:43.936194 8903 raft_consensus.cc:686] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P deacdf33fee447c88139e07de03494e7 [term 1 LEADER]: Becoming Leader. State: Replica: deacdf33fee447c88139e07de03494e7, State: Running, Role: LEADER
I20260807 07:04:43.936746 8903 consensus_queue.cc:237] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P deacdf33fee447c88139e07de03494e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e25131831d6a4c2aa8d120b6d65ff5dc" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 45467 } } peers { permanent_uuid: "e2e7caa63ae34259ba99d873914db767" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 46005 } } peers { permanent_uuid: "deacdf33fee447c88139e07de03494e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 42339 } }
I20260807 07:04:43.941947 8226 catalog_manager.cc:5495] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P deacdf33fee447c88139e07de03494e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to deacdf33fee447c88139e07de03494e7 (127.21.83.3). New cstate: current_term: 1 leader_uuid: "deacdf33fee447c88139e07de03494e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e25131831d6a4c2aa8d120b6d65ff5dc" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 45467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2e7caa63ae34259ba99d873914db767" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 46005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "deacdf33fee447c88139e07de03494e7" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 42339 } health_report { overall_health: HEALTHY } } }
I20260807 07:04:44.009763 8887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:44.003250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36896 (local address 127.21.83.3:42339)
0807 07:04:44.003479 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:44.003485 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:44.003506 (+ 21us) server_negotiation.cc:407] Connection header received
0807 07:04:44.003627 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:44.003633 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:44.003760 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:44.003928 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:04:44.004327 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:44.005419 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:44.006956 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:44.007618 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:44.007760 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:44.008320 (+ 560us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:04:44.009193 (+ 873us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:04:44.009317 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:44.009442 (+ 125us) server_negotiation.cc:299] Negotiation successful
0807 07:04:44.009523 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 07:04:44.018464 8627 raft_consensus.cc:1261] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P e2e7caa63ae34259ba99d873914db767 [term 1 FOLLOWER]: Refusing update from remote peer deacdf33fee447c88139e07de03494e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:04:44.018501 8429 raft_consensus.cc:1261] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P e25131831d6a4c2aa8d120b6d65ff5dc [term 1 FOLLOWER]: Refusing update from remote peer deacdf33fee447c88139e07de03494e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:04:44.019443 8914 consensus_queue.cc:1035] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P deacdf33fee447c88139e07de03494e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2e7caa63ae34259ba99d873914db767" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 46005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:04:44.019816 8903 consensus_queue.cc:1035] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P deacdf33fee447c88139e07de03494e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e25131831d6a4c2aa8d120b6d65ff5dc" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 45467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 07:04:44.025530 8861 tablet.cc:2369] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P deacdf33fee447c88139e07de03494e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 07:04:44.046440 8225 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:04:44.046689 8225 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:04:44.050143 8225 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 41e179b646cc45b381d9798170401005: Sending DeleteTablet for 3 replicas of tablet 5b4b2bb509514cbd9b1f57197e6e2dc7
I20260807 07:04:44.051281 8774 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b4b2bb509514cbd9b1f57197e6e2dc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:04:44 UTC) from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:36888
I20260807 07:04:44.051326 8405 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b4b2bb509514cbd9b1f57197e6e2dc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:04:44 UTC) from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:60460
I20260807 07:04:44.052224 8603 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b4b2bb509514cbd9b1f57197e6e2dc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:04:44 UTC) from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:56652
I20260807 07:04:44.056911 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 8274
I20260807 07:04:44.059769 8935 tablet_replica.cc:330] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P e2e7caa63ae34259ba99d873914db767: stopping tablet replica
I20260807 07:04:44.060288 8935 raft_consensus.cc:2229] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P e2e7caa63ae34259ba99d873914db767 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:04:44.060808 8935 raft_consensus.cc:2258] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P e2e7caa63ae34259ba99d873914db767 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:04:44.061640 8934 tablet_replica.cc:330] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P e25131831d6a4c2aa8d120b6d65ff5dc: stopping tablet replica
I20260807 07:04:44.062044 8934 raft_consensus.cc:2229] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P e25131831d6a4c2aa8d120b6d65ff5dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:04:44.062546 8934 raft_consensus.cc:2258] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P e25131831d6a4c2aa8d120b6d65ff5dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:04:44.064673 8935 ts_tablet_manager.cc:1905] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P e2e7caa63ae34259ba99d873914db767: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:04:44.071271 8934 ts_tablet_manager.cc:1905] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P e25131831d6a4c2aa8d120b6d65ff5dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:04:44.071549 8932 tablet_replica.cc:330] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P deacdf33fee447c88139e07de03494e7: stopping tablet replica
I20260807 07:04:44.072170 8932 raft_consensus.cc:2229] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P deacdf33fee447c88139e07de03494e7 [term 1 LEADER]: Raft consensus shutting down.
I20260807 07:04:44.072634 8935 ts_tablet_manager.cc:1918] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P e2e7caa63ae34259ba99d873914db767: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:04:44.072799 8935 log.cc:1196] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P e2e7caa63ae34259ba99d873914db767: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-1/wal/wals/5b4b2bb509514cbd9b1f57197e6e2dc7
I20260807 07:04:44.072993 8932 raft_consensus.cc:2258] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P deacdf33fee447c88139e07de03494e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:04:44.073392 8935 ts_tablet_manager.cc:1939] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P e2e7caa63ae34259ba99d873914db767: Deleting consensus metadata
I20260807 07:04:44.075419 8204 catalog_manager.cc:4841] TS e2e7caa63ae34259ba99d873914db767 (127.21.83.2:46005): tablet 5b4b2bb509514cbd9b1f57197e6e2dc7 (table test-table [id=6594e40bad5b464e858434a4ff2a219f]) successfully deleted
I20260807 07:04:44.077638 8932 ts_tablet_manager.cc:1905] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P deacdf33fee447c88139e07de03494e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:04:44.082854 8934 ts_tablet_manager.cc:1918] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P e25131831d6a4c2aa8d120b6d65ff5dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:04:44.083038 8934 log.cc:1196] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P e25131831d6a4c2aa8d120b6d65ff5dc: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-0/wal/wals/5b4b2bb509514cbd9b1f57197e6e2dc7
I20260807 07:04:44.083604 8934 ts_tablet_manager.cc:1939] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P e25131831d6a4c2aa8d120b6d65ff5dc: Deleting consensus metadata
I20260807 07:04:44.084151 8932 ts_tablet_manager.cc:1918] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P deacdf33fee447c88139e07de03494e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:04:44.084311 8932 log.cc:1196] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P deacdf33fee447c88139e07de03494e7: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786086078080656-21836-0/minicluster-data/ts-2/wal/wals/5b4b2bb509514cbd9b1f57197e6e2dc7
I20260807 07:04:44.084864 8932 ts_tablet_manager.cc:1939] T 5b4b2bb509514cbd9b1f57197e6e2dc7 P deacdf33fee447c88139e07de03494e7: Deleting consensus metadata
I20260807 07:04:44.085623 8204 catalog_manager.cc:4841] TS e25131831d6a4c2aa8d120b6d65ff5dc (127.21.83.1:45467): tablet 5b4b2bb509514cbd9b1f57197e6e2dc7 (table test-table [id=6594e40bad5b464e858434a4ff2a219f]) successfully deleted
I20260807 07:04:44.086467 8208 catalog_manager.cc:4841] TS deacdf33fee447c88139e07de03494e7 (127.21.83.3:42339): tablet 5b4b2bb509514cbd9b1f57197e6e2dc7 (table test-table [id=6594e40bad5b464e858434a4ff2a219f]) successfully deleted
I20260807 07:04:44.089386 8885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:44.057950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60472 (local address 127.21.83.1:45467)
0807 07:04:44.058142 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:44.058149 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:44.069215 (+ 11066us) server_negotiation.cc:407] Connection header received
0807 07:04:44.069512 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:44.069518 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:44.069606 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:44.069681 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:04:44.074351 (+ 4670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:44.075474 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:44.077829 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:44.078446 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:44.078543 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:44.082252 (+ 3709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:04:44.082266 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:04:44.082285 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:04:44.082329 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:04:44.086674 (+ 4345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:44.087502 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:44.087508 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:44.087511 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:44.087635 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:44.088001 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:44.088004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:44.088006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:44.088348 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:04:44.088569 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:44.088928 (+ 359us) server_negotiation.cc:299] Negotiation successful
0807 07:04:44.089114 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 07:04:44.090018 8490 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:04:44.180954 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 8274
I20260807 07:04:44.203078 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 8504
I20260807 07:04:44.218587 8890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:44.203943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56668 (local address 127.21.83.2:46005)
0807 07:04:44.204144 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:44.204150 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:44.204172 (+ 22us) server_negotiation.cc:407] Connection header received
0807 07:04:44.204238 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:44.204243 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:44.204314 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:44.204413 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:04:44.205876 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:44.206746 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:44.208242 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:44.208680 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:44.208801 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:44.211705 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:04:44.211722 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:04:44.211740 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:04:44.211788 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:04:44.214944 (+ 3156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:44.215828 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:44.215836 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:44.215839 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:44.216009 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:44.216439 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:44.216443 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:44.216445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:44.216874 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:04:44.217213 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:44.217768 (+ 555us) server_negotiation.cc:299] Negotiation successful
0807 07:04:44.218046 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 07:04:44.219369 8678 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:04:44.356760 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 8504
I20260807 07:04:44.378290 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 8690
I20260807 07:04:44.392562 8887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:44.379060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36910 (local address 127.21.83.3:42339)
0807 07:04:44.379274 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:44.379280 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:44.379307 (+ 27us) server_negotiation.cc:407] Connection header received
0807 07:04:44.379383 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:44.379388 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:44.379481 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:44.379600 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:04:44.381038 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:44.382061 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:44.383374 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:44.383841 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:44.383953 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:44.386565 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:04:44.386577 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:04:44.386593 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:04:44.386629 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:04:44.389714 (+ 3085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:44.390382 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:44.390389 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:44.390392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:44.390560 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:44.390928 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:44.390932 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:44.390934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:44.391324 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:04:44.391638 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:44.391967 (+ 329us) server_negotiation.cc:299] Negotiation successful
0807 07:04:44.392183 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260807 07:04:44.393294 8853 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:04:44.521224 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 8690
I20260807 07:04:44.547957 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 8178
I20260807 07:04:44.549335 8265 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:04:44.661619 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 8178
I20260807 07:04:44.689980 21836 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260807 07:04:45.035984 21836 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260807 07:04:45.036226 21836 mini_ranger.cc:66] Stopping Ranger...
I20260807 07:04:45.381249 21836 mini_ranger.cc:68] Stopped Ranger
2026-08-07 07:04:45.381 UTC [6879] LOG: received smart shutdown request
2026-08-07 07:04:45.384 UTC [6879] LOG: background worker "logical replication launcher" (PID 6890) exited with exit code 1
2026-08-07 07:04:45.384 UTC [6883] LOG: shutting down
2026-08-07 07:04:45.518 UTC [6879] LOG: database system is shut down
2026-08-07T07:04:45Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (54233 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 07:04:45 dist-test-slave-ncp9 krb5kdc[8950](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 07:04:45 dist-test-slave-ncp9 krb5kdc[8950](info): set up 2 sockets
Aug 07 07:04:45 dist-test-slave-ncp9 krb5kdc[8950](info): commencing operation
krb5kdc: starting...
W20260807 07:04:46.839293 21836 mini_kdc.cc:121] Time spent starting KDC: real 1.167s user 0.000s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Aug 07 07:04:47 dist-test-slave-ncp9 krb5kdc[8950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T07:04:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:04:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
I20260807 07:04:47.152213 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:35913
--webserver_interface=127.21.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:34965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:35913,127.21.83.61:39071,127.21.83.60:34153
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:04:47.497918 8965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:04:47.498204 8965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:04:47.498337 8965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:04:47.505568 8965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:04:47.505674 8965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:04:47.505750 8965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:04:47.505841 8965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:04:47.505896 8965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:04:47.515463 8965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:34965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:35913,127.21.83.61:39071,127.21.83.60:34153
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:35913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:04:47.516690 8965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:04:47.517931 8965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:04:47.528925 8973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:04:47.528970 8974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:04:47.529064 8976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:04:47.530100 8965 server_base.cc:1034] running on GCE node
I20260807 07:04:47.530789 8965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:04:47.532020 8965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:04:47.533226 8965 hybrid_clock.cc:648] HybridClock initialized: now 1786086287533178 us; error 39 us; skew 500 ppm
I20260807 07:04:47.533531 8965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:04:47 dist-test-slave-ncp9 krb5kdc[8950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086287, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:04:47.536445 8965 init.cc:377] Logged in from keytab as kudu/127.21.83.62@KRBTEST.COM (short username kudu)
I20260807 07:04:47.537801 8965 webserver.cc:460] Webserver started at http://127.21.83.62:43529/ using document root <none> and password file <none>
I20260807 07:04:47.538427 8965 fs_manager.cc:359] Metadata directory not provided
I20260807 07:04:47.538546 8965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:04:47.538832 8965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:04:47.541644 8965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "878c1d4f73c64186a44bc1ff70e14e6e"
format_stamp: "Formatted at 2026-08-07 07:04:47 on dist-test-slave-ncp9"
server_key: "bfae4041bad9c1e3ca33779b5c06ca47"
server_key_iv: "a9d57ca8107d4d9bcbdd84331ebcdf73"
server_key_version: "encryptionkey@0"
I20260807 07:04:47.542456 8965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "878c1d4f73c64186a44bc1ff70e14e6e"
format_stamp: "Formatted at 2026-08-07 07:04:47 on dist-test-slave-ncp9"
server_key: "bfae4041bad9c1e3ca33779b5c06ca47"
server_key_iv: "a9d57ca8107d4d9bcbdd84331ebcdf73"
server_key_version: "encryptionkey@0"
I20260807 07:04:47.547245 8965 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 07:04:47.550515 8988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:04:47.551761 8965 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 07:04:47.551910 8965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "878c1d4f73c64186a44bc1ff70e14e6e"
format_stamp: "Formatted at 2026-08-07 07:04:47 on dist-test-slave-ncp9"
server_key: "bfae4041bad9c1e3ca33779b5c06ca47"
server_key_iv: "a9d57ca8107d4d9bcbdd84331ebcdf73"
server_key_version: "encryptionkey@0"
I20260807 07:04:47.552080 8965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:04:47.573338 8965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:04:47.576805 8965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:04:47.577077 8965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:04:47.594803 8965 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.62:35913
I20260807 07:04:47.594842 9054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.62:35913 every 8 connection(s)
I20260807 07:04:47.596204 8965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:04:47.600404 9055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:04:47.601477 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 8965
I20260807 07:04:47.601687 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:04:47.602388 21836 external_mini_cluster.cc:1428] Setting key 95846a6b90f3ebc9e0195db1762ce06d
I20260807 07:04:47.607115 9055 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.83.62" port: 35913 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 07:04:47 dist-test-slave-ncp9 krb5kdc[8950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086287, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.83.61@KRBTEST.COM; defaulting to no policy
I20260807 07:04:47.629473 9057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:47.610598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:35913 (local address 127.0.0.1:37580)
0807 07:04:47.611485 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:47.611526 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 07:04:47.613726 (+ 2200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:04:47.614589 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:04:47.614596 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:04:47.615319 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:04:47.615671 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:47.615689 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:47.617284 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:47.617288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:47.618366 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:47.618378 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:47.619298 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:47.619306 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:47.619482 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:47.620085 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:04:47.620109 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:04:47.622558 (+ 2449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:04:47.625535 (+ 2977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:04:47.625547 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:04:47.625560 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:04:47.625932 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:04:47.626311 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:04:47.626316 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:04:47.626320 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:04:47.626474 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:04:47.627181 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:04:47.627189 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:04:47.627385 (+ 196us) client_negotiation.cc:769] Sending connection context
0807 07:04:47.627606 (+ 221us) client_negotiation.cc:240] Negotiation successful
0807 07:04:47.628228 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"mutex_wait_us":160,"thread_start_us":392,"threads_started":1}
I20260807 07:04:47.629508 9058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:47.612118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37580 (local address 127.21.83.62:35913)
0807 07:04:47.612879 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:47.612888 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:47.612971 (+ 83us) server_negotiation.cc:407] Connection header received
0807 07:04:47.614109 (+ 1138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:47.614115 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:47.614355 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:47.614532 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:04:47.615866 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:47.617134 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:47.618502 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:47.619054 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:47.619218 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:47.622768 (+ 3550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:04:47.622776 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:04:47.622788 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:04:47.622816 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:04:47.625423 (+ 2607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:47.626045 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:47.626051 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:47.626053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:47.626209 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:47.626571 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:47.626575 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:47.626577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:47.627043 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:04:47.627230 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:47.627735 (+ 505us) server_negotiation.cc:299] Negotiation successful
0807 07:04:47.628104 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":550,"thread_start_us":347,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.61@KRBTEST.COM" created.
I20260807 07:04:47.632649 9055 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39071 } has no permanent_uuid. Determining permanent_uuid...
I20260807 07:04:47.633710 9057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:47.633335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.61:39071 (local address 127.0.0.1:39894)
0807 07:04:47.633498 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:47.633599 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.61:39071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260807 07:04:47.633965 8995 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.83.61:39071: connect: Connection refused (error 111)
I20260807 07:04:47.635521 9057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:47.635197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.61:39071 (local address 127.0.0.1:39908)
0807 07:04:47.635346 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:47.635407 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.61:39071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260807 07:04:47.635710 9055 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.83.61:39071: Network error: Client connection negotiation failed: client connection to 127.21.83.61:39071: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.61.keytab.
Entry for principal kudu/127.21.83.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.61.keytab.
I20260807 07:04:47.662092 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.61:39071
--webserver_interface=127.21.83.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.61.keytab
--principal=kudu/127.21.83.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:34965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:35913,127.21.83.61:39071,127.21.83.60:34153
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 07:04:47.685019 9055 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39071 } attempt: 1
I20260807 07:04:47.686228 9057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:47.685788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.61:39071 (local address 127.0.0.1:39918)
0807 07:04:47.685980 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:47.686080 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.61:39071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260807 07:04:47.687191 9057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:47.686874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.61:39071 (local address 127.0.0.1:39926)
0807 07:04:47.686989 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:47.687063 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.61:39071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260807 07:04:47.687384 9055 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.83.61:39071: Network error: Client connection negotiation failed: client connection to 127.21.83.61:39071: connect: Connection refused (error 111)
I20260807 07:04:47.755633 9055 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39071 } attempt: 2
I20260807 07:04:47.756860 9057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:47.756389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.61:39071 (local address 127.0.0.1:39932)
0807 07:04:47.756625 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:47.756717 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.61:39071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260807 07:04:47.757835 9057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:47.757479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.61:39071 (local address 127.0.0.1:39936)
0807 07:04:47.757619 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:47.757703 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.61:39071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260807 07:04:47.758009 9055 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.83.61:39071: Network error: Client connection negotiation failed: client connection to 127.21.83.61:39071: connect: Connection refused (error 111)
I20260807 07:04:47.849293 9055 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39071 } attempt: 3
I20260807 07:04:47.850512 9057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:47.850055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.61:39071 (local address 127.0.0.1:39952)
0807 07:04:47.850257 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:47.850364 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.61:39071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260807 07:04:47.851507 9057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:47.851122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.61:39071 (local address 127.0.0.1:39968)
0807 07:04:47.851266 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:47.851338 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.61:39071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260807 07:04:47.851679 9055 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.83.61:39071: Network error: Client connection negotiation failed: client connection to 127.21.83.61:39071: connect: Connection refused (error 111)
I20260807 07:04:47.994958 9055 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39071 } attempt: 4
I20260807 07:04:47.996263 9057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:47.995704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.61:39071 (local address 127.0.0.1:39974)
0807 07:04:47.996023 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:47.996115 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.61:39071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176}
I20260807 07:04:47.997468 9057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:47.996963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.61:39071 (local address 127.0.0.1:39982)
0807 07:04:47.997148 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:47.997279 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.61:39071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260807 07:04:47.997768 9055 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.83.61:39071: Network error: Client connection negotiation failed: client connection to 127.21.83.61:39071: connect: Connection refused (error 111)
W20260807 07:04:48.005321 9062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:04:48.005622 9062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:04:48.005723 9062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:04:48.013134 9062 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:04:48.013236 9062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:04:48.013288 9062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:04:48.013343 9062 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:04:48.013386 9062 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:04:48.023115 9062 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:34965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:35913,127.21.83.61:39071,127.21.83.60:34153
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.61.keytab
--principal=kudu/127.21.83.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.61:39071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.83.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:04:48.024314 9062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:04:48.025555 9062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:04:48.036505 9073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:04:48.036520 9076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:04:48.036571 9074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:04:48.037569 9062 server_base.cc:1034] running on GCE node
I20260807 07:04:48.038260 9062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:04:48.039479 9062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:04:48.040678 9062 hybrid_clock.cc:648] HybridClock initialized: now 1786086288040637 us; error 29 us; skew 500 ppm
I20260807 07:04:48.041026 9062 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:04:48 dist-test-slave-ncp9 krb5kdc[8950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086288, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:04:48.043902 9062 init.cc:377] Logged in from keytab as kudu/127.21.83.61@KRBTEST.COM (short username kudu)
I20260807 07:04:48.045341 9062 webserver.cc:460] Webserver started at http://127.21.83.61:36263/ using document root <none> and password file <none>
I20260807 07:04:48.045951 9062 fs_manager.cc:359] Metadata directory not provided
I20260807 07:04:48.046060 9062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:04:48.046309 9062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:04:48.049161 9062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/minicluster-data/master-1/data/instance:
uuid: "0b5082db8f54462ebc8292beff903a70"
format_stamp: "Formatted at 2026-08-07 07:04:48 on dist-test-slave-ncp9"
server_key: "9bdc33f3e5c3e196d9d67f8bbb753e1b"
server_key_iv: "83910f637b41b4d47d8d815dff27ab7c"
server_key_version: "encryptionkey@0"
I20260807 07:04:48.049912 9062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/minicluster-data/master-1/wal/instance:
uuid: "0b5082db8f54462ebc8292beff903a70"
format_stamp: "Formatted at 2026-08-07 07:04:48 on dist-test-slave-ncp9"
server_key: "9bdc33f3e5c3e196d9d67f8bbb753e1b"
server_key_iv: "83910f637b41b4d47d8d815dff27ab7c"
server_key_version: "encryptionkey@0"
I20260807 07:04:48.054850 9062 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 07:04:48.058202 9085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:04:48.059481 9062 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 07:04:48.059635 9062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/minicluster-data/master-1/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/minicluster-data/master-1/wal
uuid: "0b5082db8f54462ebc8292beff903a70"
format_stamp: "Formatted at 2026-08-07 07:04:48 on dist-test-slave-ncp9"
server_key: "9bdc33f3e5c3e196d9d67f8bbb753e1b"
server_key_iv: "83910f637b41b4d47d8d815dff27ab7c"
server_key_version: "encryptionkey@0"
I20260807 07:04:48.059806 9062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:04:48.088615 9062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:04:48.092342 9062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:04:48.092581 9062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:04:48.110277 9062 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.61:39071
I20260807 07:04:48.110327 9149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.61:39071 every 8 connection(s)
I20260807 07:04:48.111748 9062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/minicluster-data/master-1/data/info.pb
I20260807 07:04:48.112413 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 9062
I20260807 07:04:48.112672 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/minicluster-data/master-1/wal/instance
I20260807 07:04:48.113502 21836 external_mini_cluster.cc:1428] Setting key b1f619d9cfe9cbbcf3fc55a1915f1431
I20260807 07:04:48.116021 9150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:04:48.123034 9150 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.83.62" port: 35913 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 07:04:48 dist-test-slave-ncp9 krb5kdc[8950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086288, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.61@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.83.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.60@KRBTEST.COM" created.
I20260807 07:04:48.144390 9058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:48.125985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37594 (local address 127.21.83.62:35913)
0807 07:04:48.126228 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:48.126235 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:48.127476 (+ 1241us) server_negotiation.cc:407] Connection header received
0807 07:04:48.129532 (+ 2056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:48.129539 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:48.129632 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:48.129722 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:04:48.131444 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:48.132521 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:48.134255 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:48.134881 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:48.135011 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:48.138762 (+ 3751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:04:48.138767 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:04:48.138770 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:04:48.138806 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:04:48.141456 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:48.142318 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:48.142322 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:48.142324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:48.142442 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:48.142847 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:48.142850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:48.142852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:48.143134 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:04:48.143243 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:48.143925 (+ 682us) server_negotiation.cc:299] Negotiation successful
0807 07:04:48.144097 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260807 07:04:48.145439 9153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:48.126426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:35913 (local address 127.0.0.1:37594)
0807 07:04:48.127295 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:48.127349 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0807 07:04:48.129230 (+ 1881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:04:48.129988 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:04:48.129997 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:04:48.130551 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:04:48.131243 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:48.131262 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:48.132735 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:48.132739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:48.134096 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:48.134116 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:48.135158 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:48.135167 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:48.135374 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:48.136035 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:04:48.136059 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:04:48.138503 (+ 2444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:04:48.141606 (+ 3103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:04:48.141616 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:04:48.141633 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:04:48.142211 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:04:48.142521 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:04:48.142524 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:04:48.142527 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:04:48.142768 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:04:48.143223 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:04:48.143229 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:04:48.143542 (+ 313us) client_negotiation.cc:769] Sending connection context
0807 07:04:48.143839 (+ 297us) client_negotiation.cc:240] Negotiation successful
0807 07:04:48.144285 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":356,"threads_started":1}
I20260807 07:04:48.147030 9150 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39071 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 07:04:48 dist-test-slave-ncp9 krb5kdc[8950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086288, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.61@KRBTEST.COM for kudu/127.21.83.61@KRBTEST.COM
I20260807 07:04:48.160528 9153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:48.147846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.61:39071 (local address 127.0.0.1:39990)
0807 07:04:48.148045 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:48.148062 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 07:04:48.148177 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:04:48.149071 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:04:48.149075 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:04:48.149493 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:04:48.149634 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:48.149643 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:48.150948 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:48.150953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:48.151760 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:48.151771 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:48.152597 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:48.152603 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:48.152780 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:48.153290 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:04:48.153305 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:04:48.155448 (+ 2143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:04:48.158293 (+ 2845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:04:48.158298 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:04:48.158302 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:04:48.158630 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:04:48.158956 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:04:48.158961 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:04:48.158965 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:04:48.159103 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:04:48.159650 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:04:48.159654 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:04:48.159800 (+ 146us) client_negotiation.cc:769] Sending connection context
0807 07:04:48.159911 (+ 111us) client_negotiation.cc:240] Negotiation successful
0807 07:04:48.160167 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260807 07:04:48.160650 9155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:48.147951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39990 (local address 127.21.83.61:39071)
0807 07:04:48.148684 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:48.148690 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:48.148711 (+ 21us) server_negotiation.cc:407] Connection header received
0807 07:04:48.148751 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:48.148756 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:48.148937 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:48.149060 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:04:48.149801 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:48.150819 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:48.151919 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:48.152460 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:48.152567 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:48.155636 (+ 3069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:04:48.155649 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:04:48.155663 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:04:48.155698 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:04:48.158177 (+ 2479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:48.158742 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:48.158749 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:48.158751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:48.158859 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:48.159196 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:48.159201 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:48.159204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:48.159526 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:04:48.159713 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:48.160069 (+ 356us) server_negotiation.cc:299] Negotiation successful
0807 07:04:48.160288 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":443,"threads_started":1}
I20260807 07:04:48.162868 9150 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.83.60" port: 34153 } has no permanent_uuid. Determining permanent_uuid...
I20260807 07:04:48.164007 9153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:48.163477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:34153 (local address 127.0.0.1:42232)
0807 07:04:48.163661 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:48.163771 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:34153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260807 07:04:48.164229 9089 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.83.60:34153: connect: Connection refused (error 111)
I20260807 07:04:48.165822 9153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:48.165457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:34153 (local address 127.0.0.1:42236)
0807 07:04:48.165607 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:48.165687 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:34153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260807 07:04:48.166041 9150 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.83.60:34153: Network error: Client connection negotiation failed: client connection to 127.21.83.60:34153: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.60.keytab.
Entry for principal kudu/127.21.83.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.60.keytab.
I20260807 07:04:48.176183 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.60:34153
--webserver_interface=127.21.83.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.60.keytab
--principal=kudu/127.21.83.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:34965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:35913,127.21.83.61:39071,127.21.83.60:34153
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 07:04:48.215320 9150 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.83.60" port: 34153 } attempt: 1
I20260807 07:04:48.216476 9153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:48.216027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:34153 (local address 127.0.0.1:42244)
0807 07:04:48.216229 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:48.216331 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:34153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260807 07:04:48.217463 9153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:48.217132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:34153 (local address 127.0.0.1:42252)
0807 07:04:48.217279 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:48.217359 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:34153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260807 07:04:48.217666 9150 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.83.60:34153: Network error: Client connection negotiation failed: client connection to 127.21.83.60:34153: connect: Connection refused (error 111)
I20260807 07:04:48.285866 9150 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.83.60" port: 34153 } attempt: 2
I20260807 07:04:48.286973 9153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:48.286551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:34153 (local address 127.0.0.1:42264)
0807 07:04:48.286738 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:48.286824 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:34153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260807 07:04:48.287906 9153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:48.287587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:34153 (local address 127.0.0.1:42276)
0807 07:04:48.287729 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:48.287796 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:34153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260807 07:04:48.288111 9150 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.83.60:34153: Network error: Client connection negotiation failed: client connection to 127.21.83.60:34153: connect: Connection refused (error 111)
I20260807 07:04:48.296989 9055 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39071 } attempt: 5
Aug 07 07:04:48 dist-test-slave-ncp9 krb5kdc[8950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086287, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for kudu/127.21.83.61@KRBTEST.COM
I20260807 07:04:48.311051 9057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:48.297575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.61:39071 (local address 127.0.0.1:40004)
0807 07:04:48.297764 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:48.297785 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 07:04:48.297887 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:04:48.298215 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:04:48.298218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:04:48.298615 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:04:48.298794 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:48.298803 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:48.300138 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:48.300141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:48.301085 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:48.301099 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:48.302021 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:48.302025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:48.302145 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:48.302768 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:04:48.302784 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:04:48.304730 (+ 1946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:04:48.308625 (+ 3895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:04:48.308630 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:04:48.308633 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:04:48.309107 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:04:48.309512 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:04:48.309515 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:04:48.309518 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:04:48.309645 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:04:48.310258 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:04:48.310262 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:04:48.310390 (+ 128us) client_negotiation.cc:769] Sending connection context
0807 07:04:48.310510 (+ 120us) client_negotiation.cc:240] Negotiation successful
0807 07:04:48.310781 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260807 07:04:48.311213 9155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:48.297717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40004 (local address 127.21.83.61:39071)
0807 07:04:48.297892 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:48.297897 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:48.297921 (+ 24us) server_negotiation.cc:407] Connection header received
0807 07:04:48.297982 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:48.297987 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:48.298096 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:48.298200 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:04:48.298952 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:48.300018 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:48.301215 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:48.301891 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:48.302025 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:48.304956 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:04:48.304964 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:04:48.304969 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:04:48.305009 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:04:48.308442 (+ 3433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:48.309242 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:48.309249 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:48.309252 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:48.309410 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:48.309750 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:48.309755 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:48.309759 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:48.310153 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:04:48.310298 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:48.310640 (+ 342us) server_negotiation.cc:299] Negotiation successful
0807 07:04:48.310882 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 07:04:48.312296 9055 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.83.60" port: 34153 } has no permanent_uuid. Determining permanent_uuid...
I20260807 07:04:48.313347 9057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:48.312974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:34153 (local address 127.0.0.1:42286)
0807 07:04:48.313131 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:48.313210 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:34153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260807 07:04:48.314252 9057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:48.313933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:34153 (local address 127.0.0.1:42292)
0807 07:04:48.314076 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:48.314155 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:34153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260807 07:04:48.314457 9055 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.83.60:34153: Network error: Client connection negotiation failed: client connection to 127.21.83.60:34153: connect: Connection refused (error 111)
I20260807 07:04:48.365674 9055 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.83.60" port: 34153 } attempt: 1
I20260807 07:04:48.366782 9057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:48.366314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:34153 (local address 127.0.0.1:42294)
0807 07:04:48.366502 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:48.366611 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:34153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260807 07:04:48.367765 9057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:48.367445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:34153 (local address 127.0.0.1:42306)
0807 07:04:48.367584 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:48.367651 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:34153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260807 07:04:48.367954 9055 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.83.60:34153: Network error: Client connection negotiation failed: client connection to 127.21.83.60:34153: connect: Connection refused (error 111)
I20260807 07:04:48.379292 9150 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.83.60" port: 34153 } attempt: 3
I20260807 07:04:48.380254 9153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:48.379847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:34153 (local address 127.0.0.1:42322)
0807 07:04:48.380030 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:48.380112 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:34153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260807 07:04:48.381285 9153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:48.380859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:34153 (local address 127.0.0.1:42330)
0807 07:04:48.381104 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:48.381179 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:34153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
W20260807 07:04:48.381480 9150 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.83.60:34153: Network error: Client connection negotiation failed: client connection to 127.21.83.60:34153: connect: Connection refused (error 111)
I20260807 07:04:48.436136 9055 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.83.60" port: 34153 } attempt: 2
I20260807 07:04:48.437318 9057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:48.436836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:34153 (local address 127.0.0.1:42338)
0807 07:04:48.437058 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:48.437163 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:34153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260807 07:04:48.438269 9057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:48.437951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:34153 (local address 127.0.0.1:42346)
0807 07:04:48.438097 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:48.438167 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:34153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260807 07:04:48.438470 9055 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.83.60:34153: Network error: Client connection negotiation failed: client connection to 127.21.83.60:34153: connect: Connection refused (error 111)
W20260807 07:04:48.510236 9158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:04:48.510511 9158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:04:48.510651 9158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:04:48.517638 9158 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:04:48.518095 9158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:04:48.518181 9158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:04:48.518265 9158 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:04:48.518324 9158 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:04:48.524700 9150 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.83.60" port: 34153 } attempt: 4
I20260807 07:04:48.526069 9153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:48.525446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:34153 (local address 127.0.0.1:42352)
0807 07:04:48.525752 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:48.525865 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:34153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260807 07:04:48.527223 9153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:48.526834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:34153 (local address 127.0.0.1:42366)
0807 07:04:48.526983 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:48.527083 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:34153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260807 07:04:48.527467 9150 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.83.60:34153: Network error: Client connection negotiation failed: client connection to 127.21.83.60:34153: connect: Connection refused (error 111)
I20260807 07:04:48.529853 9158 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:34965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:35913,127.21.83.61:39071,127.21.83.60:34153
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.60.keytab
--principal=kudu/127.21.83.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.60:34153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.83.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:04:48.531231 9158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:04:48.532542 9158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:04:48.544543 9165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:04:48.544615 9164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:04:48.544651 9055 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.83.60" port: 34153 } attempt: 3
W20260807 07:04:48.544484 9168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:04:48.545394 9158 server_base.cc:1034] running on GCE node
I20260807 07:04:48.545739 9057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:48.545268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:34153 (local address 127.0.0.1:42378)
0807 07:04:48.545492 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:48.545588 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:34153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260807 07:04:48.546264 9158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:04:48.546726 9057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:48.546418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:34153 (local address 127.0.0.1:42392)
0807 07:04:48.546557 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:48.546630 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:34153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260807 07:04:48.546916 9055 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.83.60:34153: Network error: Client connection negotiation failed: client connection to 127.21.83.60:34153: connect: Connection refused (error 111)
I20260807 07:04:48.547719 9158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:04:48.548954 9158 hybrid_clock.cc:648] HybridClock initialized: now 1786086288548894 us; error 45 us; skew 500 ppm
I20260807 07:04:48.549294 9158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:04:48 dist-test-slave-ncp9 krb5kdc[8950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086288, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:04:48.552256 9158 init.cc:377] Logged in from keytab as kudu/127.21.83.60@KRBTEST.COM (short username kudu)
I20260807 07:04:48.553717 9158 webserver.cc:460] Webserver started at http://127.21.83.60:38705/ using document root <none> and password file <none>
I20260807 07:04:48.554363 9158 fs_manager.cc:359] Metadata directory not provided
I20260807 07:04:48.554472 9158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:04:48.554757 9158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:04:48.557798 9158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/minicluster-data/master-2/data/instance:
uuid: "1011bccfa32f4709a0b48b2e2424d5b4"
format_stamp: "Formatted at 2026-08-07 07:04:48 on dist-test-slave-ncp9"
server_key: "1296c1fb416f370d6629c190cbcaf3c5"
server_key_iv: "4a5e136ffd482140b1bae49d73890137"
server_key_version: "encryptionkey@0"
I20260807 07:04:48.558565 9158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/minicluster-data/master-2/wal/instance:
uuid: "1011bccfa32f4709a0b48b2e2424d5b4"
format_stamp: "Formatted at 2026-08-07 07:04:48 on dist-test-slave-ncp9"
server_key: "1296c1fb416f370d6629c190cbcaf3c5"
server_key_iv: "4a5e136ffd482140b1bae49d73890137"
server_key_version: "encryptionkey@0"
I20260807 07:04:48.563328 9158 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 07:04:48.566604 9175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:04:48.567786 9158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 07:04:48.567938 9158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/minicluster-data/master-2/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/minicluster-data/master-2/wal
uuid: "1011bccfa32f4709a0b48b2e2424d5b4"
format_stamp: "Formatted at 2026-08-07 07:04:48 on dist-test-slave-ncp9"
server_key: "1296c1fb416f370d6629c190cbcaf3c5"
server_key_iv: "4a5e136ffd482140b1bae49d73890137"
server_key_version: "encryptionkey@0"
I20260807 07:04:48.568120 9158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:04:48.587824 9158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:04:48.592223 9158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:04:48.592530 9158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:04:48.611325 9158 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.60:34153
I20260807 07:04:48.611358 9243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.60:34153 every 8 connection(s)
I20260807 07:04:48.612660 9158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/minicluster-data/master-2/data/info.pb
I20260807 07:04:48.615787 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 9158
I20260807 07:04:48.615957 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786086078080656-21836-0/minicluster-data/master-2/wal/instance
I20260807 07:04:48.616178 9244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:04:48.616412 21836 external_mini_cluster.cc:1428] Setting key 38bcebd16b451d274c03ebbae1e0d9ef
I20260807 07:04:48.625105 9244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.83.62" port: 35913 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 07:04:48 dist-test-slave-ncp9 krb5kdc[8950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.60@KRBTEST.COM
Aug 07 07:04:48 dist-test-slave-ncp9 krb5kdc[8950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086288, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.60@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:04:48.638743 9249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:48.618121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42400 (local address 127.21.83.60:34153)
0807 07:04:48.618958 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:48.618972 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:48.619014 (+ 42us) server_negotiation.cc:407] Connection header received
0807 07:04:48.620509 (+ 1495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:48.620529 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:48.620871 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:48.621227 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:04:48.622986 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:48.624217 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:48.626080 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:48.626716 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:48.626845 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:48.631331 (+ 4486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:04:48.631338 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:04:48.631347 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:04:48.631385 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:04:48.634789 (+ 3404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:48.635479 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:48.635487 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:48.635498 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:48.635681 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:48.636045 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:48.636048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:48.636050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:48.636506 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:04:48.636769 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:48.637186 (+ 417us) server_negotiation.cc:299] Negotiation successful
0807 07:04:48.637549 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":380,"threads_started":1}
I20260807 07:04:48.641759 9251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:48.627661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:35913 (local address 127.0.0.1:37598)
0807 07:04:48.628340 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:48.628378 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 07:04:48.628521 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:04:48.628918 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:04:48.628925 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:04:48.629453 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:04:48.629721 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:48.629730 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:48.630745 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:48.630748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:48.631765 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:48.631777 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:48.632636 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:48.632645 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:48.632832 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:48.633327 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:04:48.633350 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:04:48.635764 (+ 2414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:04:48.639190 (+ 3426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:04:48.639201 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:04:48.639206 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:04:48.639775 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:04:48.640119 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:04:48.640122 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:04:48.640125 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:04:48.640259 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:04:48.640733 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:04:48.640740 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:04:48.640918 (+ 178us) client_negotiation.cc:769] Sending connection context
0807 07:04:48.641094 (+ 176us) client_negotiation.cc:240] Negotiation successful
0807 07:04:48.641367 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":341,"threads_started":1}
I20260807 07:04:48.641803 9058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:48.627800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37598 (local address 127.21.83.62:35913)
0807 07:04:48.627982 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:48.627988 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:48.628471 (+ 483us) server_negotiation.cc:407] Connection header received
0807 07:04:48.628659 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:48.628664 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:48.628747 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:48.628859 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:04:48.629881 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:48.630597 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:48.631938 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:48.632438 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:48.632533 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:48.635943 (+ 3410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:04:48.635949 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:04:48.635953 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:04:48.635991 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:04:48.639070 (+ 3079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:48.639870 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:48.639873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:48.639876 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:48.640002 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:48.640360 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:48.640363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:48.640366 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:48.640609 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:04:48.640745 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:48.641213 (+ 468us) server_negotiation.cc:299] Negotiation successful
0807 07:04:48.641412 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 07:04:48.643487 9244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39071 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 07:04:48 dist-test-slave-ncp9 krb5kdc[8950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
Aug 07 07:04:48 dist-test-slave-ncp9 krb5kdc[8950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086288, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.60@KRBTEST.COM for kudu/127.21.83.61@KRBTEST.COM
I20260807 07:04:48.656587 9251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:48.644237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.61:39071 (local address 127.0.0.1:40012)
0807 07:04:48.644392 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:48.644408 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 07:04:48.644501 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:04:48.644786 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:04:48.644790 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:04:48.645156 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:04:48.645286 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:48.645294 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:48.646245 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:48.646251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:48.647074 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:48.647094 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:48.647747 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:48.647753 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:48.647920 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:48.648395 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:04:48.648409 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:04:48.650619 (+ 2210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:04:48.654006 (+ 3387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:04:48.654011 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:04:48.654014 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:04:48.654483 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:04:48.654870 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:04:48.654875 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:04:48.654878 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:04:48.655052 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:04:48.655598 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:04:48.655602 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:04:48.655751 (+ 149us) client_negotiation.cc:769] Sending connection context
0807 07:04:48.655894 (+ 143us) client_negotiation.cc:240] Negotiation successful
0807 07:04:48.656157 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260807 07:04:48.656572 9058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:48.641526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37602 (local address 127.21.83.62:35913)
0807 07:04:48.642072 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:48.642077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:48.642096 (+ 19us) server_negotiation.cc:407] Connection header received
0807 07:04:48.642141 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:48.642146 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:48.642228 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:48.642314 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:04:48.643764 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:48.644697 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:48.645924 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:48.646335 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:48.646443 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:48.650591 (+ 4148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:04:48.650595 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:04:48.650598 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:04:48.650627 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:04:48.653764 (+ 3137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:48.654511 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:48.654515 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:48.654517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:48.654679 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:48.655062 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:48.655067 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:48.655071 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:48.655433 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:04:48.655709 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:48.656030 (+ 321us) server_negotiation.cc:299] Negotiation successful
0807 07:04:48.656231 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":463,"threads_started":1}
I20260807 07:04:48.656648 9155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:48.644170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40012 (local address 127.21.83.61:39071)
0807 07:04:48.644327 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:48.644332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:48.644463 (+ 131us) server_negotiation.cc:407] Connection header received
0807 07:04:48.644610 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:48.644613 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:48.644675 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:48.644752 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:04:48.645421 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:48.646122 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:48.647207 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:48.647613 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:48.647716 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:48.650796 (+ 3080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:04:48.650803 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:04:48.650807 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:04:48.650845 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:04:48.653868 (+ 3023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:48.654594 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:48.654600 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:48.654603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:48.654768 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:48.655141 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:48.655146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:48.655149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:48.655469 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:04:48.655593 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:48.656017 (+ 424us) server_negotiation.cc:299] Negotiation successful
0807 07:04:48.656299 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260807 07:04:48.657543 9244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.83.60" port: 34153 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 07:04:48 dist-test-slave-ncp9 krb5kdc[8950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086288, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.60@KRBTEST.COM for kudu/127.21.83.60@KRBTEST.COM
Aug 07 07:04:48 dist-test-slave-ncp9 krb5kdc[8950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.61@KRBTEST.COM
I20260807 07:04:48.669713 9251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:48.658138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:34153 (local address 127.0.0.1:42404)
0807 07:04:48.658276 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:48.658294 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 07:04:48.658381 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:04:48.658687 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:04:48.658690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:04:48.659040 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:04:48.659179 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:48.659187 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:48.660078 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:48.660083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:48.660941 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:48.660957 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:48.661626 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:48.661631 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:48.661795 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:48.662292 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:04:48.662308 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:04:48.664628 (+ 2320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:04:48.667522 (+ 2894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:04:48.667526 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:04:48.667529 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:04:48.667860 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:04:48.668237 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:04:48.668242 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:04:48.668245 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:04:48.668391 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:04:48.668994 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:04:48.668997 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:04:48.669125 (+ 128us) client_negotiation.cc:769] Sending connection context
0807 07:04:48.669241 (+ 116us) client_negotiation.cc:240] Negotiation successful
0807 07:04:48.669430 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260807 07:04:48.670081 9249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:48.658331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42404 (local address 127.21.83.60:34153)
0807 07:04:48.658466 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:48.658471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:48.658486 (+ 15us) server_negotiation.cc:407] Connection header received
0807 07:04:48.658522 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:48.658525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:48.658581 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:48.658654 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:04:48.659305 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:48.659956 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:48.661076 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:48.661484 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:48.661598 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:48.664790 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:04:48.664794 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:04:48.664797 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:04:48.664825 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:04:48.667407 (+ 2582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:48.667971 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:48.667977 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:48.667981 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:48.668131 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:48.668494 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:48.668499 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:48.668502 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:48.668857 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:04:48.669035 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:48.669363 (+ 328us) server_negotiation.cc:299] Negotiation successful
0807 07:04:48.669617 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260807 07:04:48.672358 9155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:48.658177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40020 (local address 127.21.83.61:39071)
0807 07:04:48.658318 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:48.658323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:48.658479 (+ 156us) server_negotiation.cc:407] Connection header received
0807 07:04:48.658617 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:48.658622 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:48.658681 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:48.658752 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:04:48.659846 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:48.660507 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:48.661762 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:48.662165 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:48.662281 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:48.666879 (+ 4598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:04:48.666883 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:04:48.666887 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:04:48.666914 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:04:48.669822 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:48.670507 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:48.670511 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:48.670513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:48.670644 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:48.671088 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:48.671093 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:48.671096 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:48.671374 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:04:48.671600 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:48.671920 (+ 320us) server_negotiation.cc:299] Negotiation successful
0807 07:04:48.672078 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260807 07:04:48.676199 9244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1011bccfa32f4709a0b48b2e2424d5b4: Bootstrap starting.
I20260807 07:04:48.677006 21836 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 07:04:48.679234 9244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1011bccfa32f4709a0b48b2e2424d5b4: Neither blocks nor log segments found. Creating new log.
I20260807 07:04:48.680347 9244 log.cc:824] T 00000000000000000000000000000000 P 1011bccfa32f4709a0b48b2e2424d5b4: Log is configured to *not* fsync() on all Append() calls
I20260807 07:04:48.683563 9244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1011bccfa32f4709a0b48b2e2424d5b4: No bootstrap required, opened a new log
I20260807 07:04:48.693835 9244 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1011bccfa32f4709a0b48b2e2424d5b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "878c1d4f73c64186a44bc1ff70e14e6e" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 35913 } } peers { permanent_uuid: "0b5082db8f54462ebc8292beff903a70" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39071 } } peers { permanent_uuid: "1011bccfa32f4709a0b48b2e2424d5b4" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 34153 } }
I20260807 07:04:48.694357 9244 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1011bccfa32f4709a0b48b2e2424d5b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:04:48.694535 9244 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1011bccfa32f4709a0b48b2e2424d5b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1011bccfa32f4709a0b48b2e2424d5b4, State: Initialized, Role: FOLLOWER
I20260807 07:04:48.695325 9244 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1011bccfa32f4709a0b48b2e2424d5b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "878c1d4f73c64186a44bc1ff70e14e6e" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 35913 } } peers { permanent_uuid: "0b5082db8f54462ebc8292beff903a70" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39071 } } peers { permanent_uuid: "1011bccfa32f4709a0b48b2e2424d5b4" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 34153 } }
I20260807 07:04:48.702675 9263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1011bccfa32f4709a0b48b2e2424d5b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "878c1d4f73c64186a44bc1ff70e14e6e" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 35913 } } peers { permanent_uuid: "0b5082db8f54462ebc8292beff903a70" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39071 } } peers { permanent_uuid: "1011bccfa32f4709a0b48b2e2424d5b4" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 34153 } } }
I20260807 07:04:48.704793 9058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:48.682162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37610 (local address 127.21.83.62:35913)
0807 07:04:48.682594 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:48.682601 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:48.686034 (+ 3433us) server_negotiation.cc:407] Connection header received
0807 07:04:48.686213 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:48.686218 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:48.686290 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:48.686384 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:04:48.688477 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:48.689571 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:48.691686 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:48.692346 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:48.692437 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:48.696397 (+ 3960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:04:48.696404 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:04:48.696410 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:04:48.696463 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:04:48.700734 (+ 4271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:48.702004 (+ 1270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:48.702011 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:48.702014 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:48.702243 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:48.702834 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:48.702839 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:48.702842 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:48.703222 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:04:48.703472 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:48.704139 (+ 667us) server_negotiation.cc:299] Negotiation successful
0807 07:04:48.704419 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293}
I20260807 07:04:48.705255 9155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:48.682382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40024 (local address 127.21.83.61:39071)
0807 07:04:48.682735 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:48.682742 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:48.685497 (+ 2755us) server_negotiation.cc:407] Connection header received
0807 07:04:48.686438 (+ 941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:48.686443 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:48.686529 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:48.686607 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:04:48.689110 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:48.690289 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:48.692796 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:48.693547 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:48.693651 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:48.697793 (+ 4142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:04:48.697801 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:04:48.697806 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:04:48.697860 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:04:48.701673 (+ 3813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:48.702767 (+ 1094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:48.702774 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:48.702777 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:48.702952 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:48.703455 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:48.703460 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:48.703464 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:48.703830 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:04:48.704077 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:48.704607 (+ 530us) server_negotiation.cc:299] Negotiation successful
0807 07:04:48.704873 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239}
I20260807 07:04:48.705541 9263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1011bccfa32f4709a0b48b2e2424d5b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 07:04:48.705407 9244 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1011bccfa32f4709a0b48b2e2424d5b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:04:48.715040 9249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:48.685771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58250 (local address 127.21.83.60:34153)
0807 07:04:48.685973 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:48.685983 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:48.697899 (+ 11916us) server_negotiation.cc:407] Connection header received
0807 07:04:48.698346 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:48.698353 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:48.698456 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:48.698539 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:04:48.700470 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:48.701868 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:48.704083 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:48.704844 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:48.704972 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:48.708492 (+ 3520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:04:48.708498 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:04:48.708501 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:04:48.708539 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:04:48.711674 (+ 3135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:48.712556 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:48.712562 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:48.712565 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:48.712761 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:48.713217 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:48.713222 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:48.713225 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:48.713529 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:04:48.713780 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:48.714245 (+ 465us) server_negotiation.cc:299] Negotiation successful
0807 07:04:48.714488 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260807 07:04:48.718022 9276 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1011bccfa32f4709a0b48b2e2424d5b4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 07:04:48.718182 9276 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 07:04:48.724113 9055 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.83.60" port: 34153 } attempt: 4
I20260807 07:04:48.715101 9158 master_runner.cc:186] Error getting master registration for 127.21.83.62:35913: OK, instance_id { permanent_uuid: "878c1d4f73c64186a44bc1ff70e14e6e" instance_seqno: 1786086287589395 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 07:04:48.725009 9158 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Aug 07 07:04:48 dist-test-slave-ncp9 krb5kdc[8950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086287, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for kudu/127.21.83.60@KRBTEST.COM
I20260807 07:04:48.739403 9057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:48.724631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:34153 (local address 127.0.0.1:58252)
0807 07:04:48.724928 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:48.724947 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 07:04:48.725061 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:04:48.725355 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:04:48.725359 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:04:48.725801 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:04:48.726016 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:48.726031 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:48.727508 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:48.727514 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:48.728977 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:48.728994 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:48.729725 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:48.729732 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:48.729917 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:48.730537 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:04:48.730562 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:04:48.732997 (+ 2435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:04:48.736851 (+ 3854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:04:48.736855 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:04:48.736858 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:04:48.737259 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:04:48.737677 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:04:48.737680 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:04:48.737682 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:04:48.737801 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:04:48.738434 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:04:48.738439 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:04:48.738602 (+ 163us) client_negotiation.cc:769] Sending connection context
0807 07:04:48.738742 (+ 140us) client_negotiation.cc:240] Negotiation successful
0807 07:04:48.739029 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20260807 07:04:48.739486 9249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:48.724719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58252 (local address 127.21.83.60:34153)
0807 07:04:48.724930 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:48.724935 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:48.725012 (+ 77us) server_negotiation.cc:407] Connection header received
0807 07:04:48.725171 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:48.725176 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:48.725232 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:48.725307 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:04:48.726179 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:48.727329 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:48.729124 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:48.729573 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:48.729673 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:48.733218 (+ 3545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:04:48.733224 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:04:48.733229 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:04:48.733264 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:04:48.736726 (+ 3462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:48.737395 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:48.737400 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:48.737404 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:48.737582 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:48.737904 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:48.737908 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:48.737912 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:48.738300 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:04:48.738459 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:48.738875 (+ 416us) server_negotiation.cc:299] Negotiation successful
0807 07:04:48.739128 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260807 07:04:48.745440 9055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 878c1d4f73c64186a44bc1ff70e14e6e: Bootstrap starting.
I20260807 07:04:48.749500 9055 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 878c1d4f73c64186a44bc1ff70e14e6e: Neither blocks nor log segments found. Creating new log.
I20260807 07:04:48.750801 9055 log.cc:824] T 00000000000000000000000000000000 P 878c1d4f73c64186a44bc1ff70e14e6e: Log is configured to *not* fsync() on all Append() calls
I20260807 07:04:48.754094 9055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 878c1d4f73c64186a44bc1ff70e14e6e: No bootstrap required, opened a new log
I20260807 07:04:48.765383 9055 raft_consensus.cc:348] T 00000000000000000000000000000000 P 878c1d4f73c64186a44bc1ff70e14e6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "878c1d4f73c64186a44bc1ff70e14e6e" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 35913 } } peers { permanent_uuid: "0b5082db8f54462ebc8292beff903a70" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39071 } } peers { permanent_uuid: "1011bccfa32f4709a0b48b2e2424d5b4" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 34153 } }
I20260807 07:04:48.765844 9055 raft_consensus.cc:374] T 00000000000000000000000000000000 P 878c1d4f73c64186a44bc1ff70e14e6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:04:48.765981 9055 raft_consensus.cc:729] T 00000000000000000000000000000000 P 878c1d4f73c64186a44bc1ff70e14e6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 878c1d4f73c64186a44bc1ff70e14e6e, State: Initialized, Role: FOLLOWER
I20260807 07:04:48.766705 9055 consensus_queue.cc:260] T 00000000000000000000000000000000 P 878c1d4f73c64186a44bc1ff70e14e6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "878c1d4f73c64186a44bc1ff70e14e6e" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 35913 } } peers { permanent_uuid: "0b5082db8f54462ebc8292beff903a70" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39071 } } peers { permanent_uuid: "1011bccfa32f4709a0b48b2e2424d5b4" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 34153 } }
I20260807 07:04:48.767684 9279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 878c1d4f73c64186a44bc1ff70e14e6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "878c1d4f73c64186a44bc1ff70e14e6e" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 35913 } } peers { permanent_uuid: "0b5082db8f54462ebc8292beff903a70" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39071 } } peers { permanent_uuid: "1011bccfa32f4709a0b48b2e2424d5b4" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 34153 } } }
I20260807 07:04:48.768025 9279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 878c1d4f73c64186a44bc1ff70e14e6e [sys.catalog]: This master's current role is: FOLLOWER
I20260807 07:04:48.769204 9055 sys_catalog.cc:564] T 00000000000000000000000000000000 P 878c1d4f73c64186a44bc1ff70e14e6e [sys.catalog]: configured and running, proceeding with master startup.
W20260807 07:04:48.780125 9297 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 878c1d4f73c64186a44bc1ff70e14e6e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 07:04:48.780323 9297 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 07:04:48.779821 8965 master_runner.cc:186] Error getting master registration for 127.21.83.61:39071: OK, instance_id { permanent_uuid: "0b5082db8f54462ebc8292beff903a70" instance_seqno: 1786086288104622 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 07:04:48.788817 8965 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 07:04:48.797879 9263 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1011bccfa32f4709a0b48b2e2424d5b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:04:48.798198 9263 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1011bccfa32f4709a0b48b2e2424d5b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "878c1d4f73c64186a44bc1ff70e14e6e" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 35913 } } peers { permanent_uuid: "0b5082db8f54462ebc8292beff903a70" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39071 } } peers { permanent_uuid: "1011bccfa32f4709a0b48b2e2424d5b4" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 34153 } }
I20260807 07:04:48.799896 9263 leader_election.cc:288] T 00000000000000000000000000000000 P 1011bccfa32f4709a0b48b2e2424d5b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 878c1d4f73c64186a44bc1ff70e14e6e (127.21.83.62:35913), 0b5082db8f54462ebc8292beff903a70 (127.21.83.61:39071)
I20260807 07:04:48.801193 9024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1011bccfa32f4709a0b48b2e2424d5b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "878c1d4f73c64186a44bc1ff70e14e6e" is_pre_election: true
I20260807 07:04:48.801815 9024 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 878c1d4f73c64186a44bc1ff70e14e6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1011bccfa32f4709a0b48b2e2424d5b4 in term 0.
I20260807 07:04:48.802670 9177 leader_election.cc:302] T 00000000000000000000000000000000 P 1011bccfa32f4709a0b48b2e2424d5b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1011bccfa32f4709a0b48b2e2424d5b4, 878c1d4f73c64186a44bc1ff70e14e6e; no voters:
I20260807 07:04:48.803081 9263 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1011bccfa32f4709a0b48b2e2424d5b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 07:04:48.803215 9263 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1011bccfa32f4709a0b48b2e2424d5b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:04:48.803321 9263 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1011bccfa32f4709a0b48b2e2424d5b4 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:04:48.806010 9263 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1011bccfa32f4709a0b48b2e2424d5b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "878c1d4f73c64186a44bc1ff70e14e6e" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 35913 } } peers { permanent_uuid: "0b5082db8f54462ebc8292beff903a70" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39071 } } peers { permanent_uuid: "1011bccfa32f4709a0b48b2e2424d5b4" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 34153 } }
I20260807 07:04:48.806988 9263 leader_election.cc:288] T 00000000000000000000000000000000 P 1011bccfa32f4709a0b48b2e2424d5b4 [CANDIDATE]: Term 1 election: Requested vote from peers 878c1d4f73c64186a44bc1ff70e14e6e (127.21.83.62:35913), 0b5082db8f54462ebc8292beff903a70 (127.21.83.61:39071)
I20260807 07:04:48.807159 9024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1011bccfa32f4709a0b48b2e2424d5b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "878c1d4f73c64186a44bc1ff70e14e6e"
I20260807 07:04:48.807363 9024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 878c1d4f73c64186a44bc1ff70e14e6e [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:04:48.810623 9024 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 878c1d4f73c64186a44bc1ff70e14e6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1011bccfa32f4709a0b48b2e2424d5b4 in term 1.
I20260807 07:04:48.811029 9177 leader_election.cc:302] T 00000000000000000000000000000000 P 1011bccfa32f4709a0b48b2e2424d5b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1011bccfa32f4709a0b48b2e2424d5b4, 878c1d4f73c64186a44bc1ff70e14e6e; no voters:
I20260807 07:04:48.811336 9263 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1011bccfa32f4709a0b48b2e2424d5b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:04:48.812057 9263 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1011bccfa32f4709a0b48b2e2424d5b4 [term 1 LEADER]: Becoming Leader. State: Replica: 1011bccfa32f4709a0b48b2e2424d5b4, State: Running, Role: LEADER
I20260807 07:04:48.812724 9263 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1011bccfa32f4709a0b48b2e2424d5b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "878c1d4f73c64186a44bc1ff70e14e6e" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 35913 } } peers { permanent_uuid: "0b5082db8f54462ebc8292beff903a70" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39071 } } peers { permanent_uuid: "1011bccfa32f4709a0b48b2e2424d5b4" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 34153 } }
I20260807 07:04:48.816383 9299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1011bccfa32f4709a0b48b2e2424d5b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1011bccfa32f4709a0b48b2e2424d5b4. Latest consensus state: current_term: 1 leader_uuid: "1011bccfa32f4709a0b48b2e2424d5b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "878c1d4f73c64186a44bc1ff70e14e6e" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 35913 } } peers { permanent_uuid: "0b5082db8f54462ebc8292beff903a70" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39071 } } peers { permanent_uuid: "1011bccfa32f4709a0b48b2e2424d5b4" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 34153 } } }
I20260807 07:04:48.816828 9299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1011bccfa32f4709a0b48b2e2424d5b4 [sys.catalog]: This master's current role is: LEADER
I20260807 07:04:48.826683 9150 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.83.60" port: 34153 } attempt: 5
I20260807 07:04:48.829084 9301 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260807 07:04:48.829792 8998 tablet.cc:2369] T 00000000000000000000000000000000 P 878c1d4f73c64186a44bc1ff70e14e6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 07:04:48.831545 9301 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 07 07:04:48 dist-test-slave-ncp9 krb5kdc[8950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086288, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.61@KRBTEST.COM for kudu/127.21.83.60@KRBTEST.COM
W20260807 07:04:48.846077 9185 tablet.cc:2369] T 00000000000000000000000000000000 P 1011bccfa32f4709a0b48b2e2424d5b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 07:04:48.847774 9024 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 878c1d4f73c64186a44bc1ff70e14e6e [term 1 FOLLOWER]: Refusing update from remote peer 1011bccfa32f4709a0b48b2e2424d5b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:04:48.848944 9299 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1011bccfa32f4709a0b48b2e2424d5b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "878c1d4f73c64186a44bc1ff70e14e6e" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 35913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:04:48.851487 9153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:48.827265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:34153 (local address 127.0.0.1:58256)
0807 07:04:48.827612 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:48.827632 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 07:04:48.827745 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:04:48.828608 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:04:48.828613 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:04:48.829185 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:04:48.829421 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:48.829435 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:48.831950 (+ 2515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:48.831955 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:48.836561 (+ 4606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:48.836578 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:48.837871 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:48.837878 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:48.838109 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:48.838788 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:04:48.838810 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:04:48.841638 (+ 2828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:04:48.847294 (+ 5656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:04:48.847299 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:04:48.847304 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:04:48.847813 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:04:48.848534 (+ 721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:04:48.848538 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:04:48.848541 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:04:48.848742 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:04:48.850527 (+ 1785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:04:48.850533 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:04:48.850709 (+ 176us) client_negotiation.cc:769] Sending connection context
0807 07:04:48.850851 (+ 142us) client_negotiation.cc:240] Negotiation successful
0807 07:04:48.851133 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215}
I20260807 07:04:48.853351 9249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:48.827946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58256 (local address 127.21.83.60:34153)
0807 07:04:48.828141 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:48.828147 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:48.828165 (+ 18us) server_negotiation.cc:407] Connection header received
0807 07:04:48.828224 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:48.828229 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:48.828319 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:48.828411 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:04:48.830600 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:48.831715 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:48.836745 (+ 5030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:48.837510 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:48.837629 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:48.841856 (+ 4227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:04:48.841862 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:04:48.841867 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:04:48.841903 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:04:48.845765 (+ 3862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:48.848068 (+ 2303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:48.848073 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:48.848076 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:48.848340 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:48.849636 (+ 1296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:48.849641 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:48.849644 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:48.850338 (+ 694us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:04:48.851924 (+ 1586us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:48.852176 (+ 252us) server_negotiation.cc:299] Negotiation successful
0807 07:04:48.852580 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 07:04:48.860459 9150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b5082db8f54462ebc8292beff903a70: Bootstrap starting.
I20260807 07:04:48.865484 9150 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b5082db8f54462ebc8292beff903a70: Neither blocks nor log segments found. Creating new log.
I20260807 07:04:48.866868 9150 log.cc:824] T 00000000000000000000000000000000 P 0b5082db8f54462ebc8292beff903a70: Log is configured to *not* fsync() on all Append() calls
I20260807 07:04:48.873879 9150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b5082db8f54462ebc8292beff903a70: No bootstrap required, opened a new log
I20260807 07:04:48.886044 9279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 878c1d4f73c64186a44bc1ff70e14e6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1011bccfa32f4709a0b48b2e2424d5b4. Latest consensus state: current_term: 1 leader_uuid: "1011bccfa32f4709a0b48b2e2424d5b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "878c1d4f73c64186a44bc1ff70e14e6e" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 35913 } } peers { permanent_uuid: "0b5082db8f54462ebc8292beff903a70" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39071 } } peers { permanent_uuid: "1011bccfa32f4709a0b48b2e2424d5b4" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 34153 } } }
I20260807 07:04:48.886399 9279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 878c1d4f73c64186a44bc1ff70e14e6e [sys.catalog]: This master's current role is: FOLLOWER
I20260807 07:04:48.890357 9302 mvcc.cc:205] Tried to move back new op lower bound from 7315809439104757760 to 7315809438981910528. Current Snapshot: MvccSnapshot[applied={T|T < 7315809439104757760}]
I20260807 07:04:48.894294 9303 mvcc.cc:205] Tried to move back new op lower bound from 7315809439104757760 to 7315809438981910528. Current Snapshot: MvccSnapshot[applied={T|T < 7315809439104757760}]
I20260807 07:04:48.900269 9150 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b5082db8f54462ebc8292beff903a70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "878c1d4f73c64186a44bc1ff70e14e6e" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 35913 } } peers { permanent_uuid: "0b5082db8f54462ebc8292beff903a70" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39071 } } peers { permanent_uuid: "1011bccfa32f4709a0b48b2e2424d5b4" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 34153 } }
I20260807 07:04:48.904754 9279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 878c1d4f73c64186a44bc1ff70e14e6e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1011bccfa32f4709a0b48b2e2424d5b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "878c1d4f73c64186a44bc1ff70e14e6e" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 35913 } } peers { permanent_uuid: "0b5082db8f54462ebc8292beff903a70" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39071 } } peers { permanent_uuid: "1011bccfa32f4709a0b48b2e2424d5b4" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 34153 } } }
I20260807 07:04:48.905022 9279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 878c1d4f73c64186a44bc1ff70e14e6e [sys.catalog]: This master's current role is: FOLLOWER
I20260807 07:04:48.909235 9299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1011bccfa32f4709a0b48b2e2424d5b4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1011bccfa32f4709a0b48b2e2424d5b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "878c1d4f73c64186a44bc1ff70e14e6e" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 35913 } } peers { permanent_uuid: "0b5082db8f54462ebc8292beff903a70" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39071 } } peers { permanent_uuid: "1011bccfa32f4709a0b48b2e2424d5b4" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 34153 } } }
I20260807 07:04:48.909526 9299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1011bccfa32f4709a0b48b2e2424d5b4 [sys.catalog]: This master's current role is: LEADER
I20260807 07:04:48.909803 9150 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b5082db8f54462ebc8292beff903a70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:04:48.909992 9150 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b5082db8f54462ebc8292beff903a70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b5082db8f54462ebc8292beff903a70, State: Initialized, Role: FOLLOWER
I20260807 07:04:48.911098 9150 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b5082db8f54462ebc8292beff903a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "878c1d4f73c64186a44bc1ff70e14e6e" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 35913 } } peers { permanent_uuid: "0b5082db8f54462ebc8292beff903a70" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39071 } } peers { permanent_uuid: "1011bccfa32f4709a0b48b2e2424d5b4" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 34153 } }
I20260807 07:04:48.911968 9301 catalog_manager.cc:1347] Generated new cluster ID: 3720727fee6d4dc79605773f3d7a5bbd
I20260807 07:04:48.912065 9301 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:04:48.915711 9310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b5082db8f54462ebc8292beff903a70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "878c1d4f73c64186a44bc1ff70e14e6e" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 35913 } } peers { permanent_uuid: "0b5082db8f54462ebc8292beff903a70" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39071 } } peers { permanent_uuid: "1011bccfa32f4709a0b48b2e2424d5b4" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 34153 } } }
I20260807 07:04:48.917618 9310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b5082db8f54462ebc8292beff903a70 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 07:04:48.917497 9150 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b5082db8f54462ebc8292beff903a70 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:04:48.917827 9115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b5082db8f54462ebc8292beff903a70 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:04:48.921103 9116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1011bccfa32f4709a0b48b2e2424d5b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b5082db8f54462ebc8292beff903a70"
I20260807 07:04:48.922963 9118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1011bccfa32f4709a0b48b2e2424d5b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b5082db8f54462ebc8292beff903a70" is_pre_election: true
I20260807 07:04:48.925174 9115 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0b5082db8f54462ebc8292beff903a70 [term 1 FOLLOWER]: Refusing update from remote peer 1011bccfa32f4709a0b48b2e2424d5b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:04:48.926167 9263 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1011bccfa32f4709a0b48b2e2424d5b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b5082db8f54462ebc8292beff903a70" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:04:48.954383 9310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b5082db8f54462ebc8292beff903a70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1011bccfa32f4709a0b48b2e2424d5b4. Latest consensus state: current_term: 1 leader_uuid: "1011bccfa32f4709a0b48b2e2424d5b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "878c1d4f73c64186a44bc1ff70e14e6e" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 35913 } } peers { permanent_uuid: "0b5082db8f54462ebc8292beff903a70" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39071 } } peers { permanent_uuid: "1011bccfa32f4709a0b48b2e2424d5b4" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 34153 } } }
I20260807 07:04:48.954737 9310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b5082db8f54462ebc8292beff903a70 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 07:04:48.955116 9315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b5082db8f54462ebc8292beff903a70 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1011bccfa32f4709a0b48b2e2424d5b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "878c1d4f73c64186a44bc1ff70e14e6e" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 35913 } } peers { permanent_uuid: "0b5082db8f54462ebc8292beff903a70" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39071 } } peers { permanent_uuid: "1011bccfa32f4709a0b48b2e2424d5b4" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 34153 } } }
I20260807 07:04:48.955577 9315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b5082db8f54462ebc8292beff903a70 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 07:04:48.960697 9263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1011bccfa32f4709a0b48b2e2424d5b4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1011bccfa32f4709a0b48b2e2424d5b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "878c1d4f73c64186a44bc1ff70e14e6e" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 35913 } } peers { permanent_uuid: "0b5082db8f54462ebc8292beff903a70" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39071 } } peers { permanent_uuid: "1011bccfa32f4709a0b48b2e2424d5b4" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 34153 } } }
I20260807 07:04:48.961053 9263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1011bccfa32f4709a0b48b2e2424d5b4 [sys.catalog]: This master's current role is: LEADER
I20260807 07:04:48.970727 9301 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:04:48.972445 9301 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:04:49.020720 9301 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1011bccfa32f4709a0b48b2e2424d5b4: Generated new TSK 0
I20260807 07:04:49.022133 9301 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 07:04:49.034343 9329 catalog_manager.cc:1259] Loaded cluster ID: 3720727fee6d4dc79605773f3d7a5bbd
I20260807 07:04:49.034449 9329 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0b5082db8f54462ebc8292beff903a70: loading cluster ID for follower catalog manager: success
I20260807 07:04:49.058275 9329 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0b5082db8f54462ebc8292beff903a70: acquiring CA information for follower catalog manager: success
I20260807 07:04:49.064417 9329 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0b5082db8f54462ebc8292beff903a70: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 07:04:49.782405 9297 catalog_manager.cc:1259] Loaded cluster ID: 3720727fee6d4dc79605773f3d7a5bbd
I20260807 07:04:49.782522 9297 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 878c1d4f73c64186a44bc1ff70e14e6e: loading cluster ID for follower catalog manager: success
I20260807 07:04:49.785048 9297 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 878c1d4f73c64186a44bc1ff70e14e6e: acquiring CA information for follower catalog manager: success
I20260807 07:04:49.787149 9297 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 878c1d4f73c64186a44bc1ff70e14e6e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 07:04:50.177214 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 8965
I20260807 07:04:50.178234 9047 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:04:50.286518 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 8965
I20260807 07:04:50.313388 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 9062
I20260807 07:04:50.314538 9144 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:04:50.422832 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 9062
I20260807 07:04:50.445518 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 9158
I20260807 07:04:50.446601 9239 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:04:50.555544 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 9158
2026-08-07T07:04:50Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4931 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786086078080656-21836-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 07:04:50 dist-test-slave-ncp9 krb5kdc[9346](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 07:04:50 dist-test-slave-ncp9 krb5kdc[9346](info): set up 2 sockets
Aug 07 07:04:50 dist-test-slave-ncp9 krb5kdc[9346](info): commencing operation
krb5kdc: starting...
W20260807 07:04:51.765450 21836 mini_kdc.cc:121] Time spent starting KDC: real 1.162s user 0.003s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Aug 07 07:04:51 dist-test-slave-ncp9 krb5kdc[9346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T07:04:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:04:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
I20260807 07:04:52.069948 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:34697
--webserver_interface=127.21.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:36159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:34697
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:04:52.407128 9366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:04:52.407408 9366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:04:52.407512 9366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:04:52.414969 9366 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:04:52.415114 9366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:04:52.415176 9366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:04:52.415261 9366 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:04:52.415328 9366 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:04:52.425997 9366 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:36159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:34697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:34697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:04:52.427420 9366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:04:52.428854 9366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:04:52.441318 9372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:04:52.441382 9376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:04:52.441350 9374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:04:52.442317 9366 server_base.cc:1034] running on GCE node
I20260807 07:04:52.443240 9366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:04:52.444854 9366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:04:52.446103 9366 hybrid_clock.cc:648] HybridClock initialized: now 1786086292446039 us; error 52 us; skew 500 ppm
I20260807 07:04:52.446471 9366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:04:52 dist-test-slave-ncp9 krb5kdc[9346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086292, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:04:52.453511 9366 init.cc:377] Logged in from keytab as kudu/127.21.83.62@KRBTEST.COM (short username kudu)
I20260807 07:04:52.455163 9366 webserver.cc:460] Webserver started at http://127.21.83.62:38139/ using document root <none> and password file <none>
I20260807 07:04:52.455899 9366 fs_manager.cc:359] Metadata directory not provided
I20260807 07:04:52.456008 9366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:04:52.456281 9366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:04:52.459314 9366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "b81230f1b6424b689ebb5ca52fa568b7"
format_stamp: "Formatted at 2026-08-07 07:04:52 on dist-test-slave-ncp9"
server_key: "1df82ac05975876fa887abc74fe700d7"
server_key_iv: "2eb6fe5a2ddd1c8cf1e7617eea9b285b"
server_key_version: "encryptionkey@0"
I20260807 07:04:52.460014 9366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "b81230f1b6424b689ebb5ca52fa568b7"
format_stamp: "Formatted at 2026-08-07 07:04:52 on dist-test-slave-ncp9"
server_key: "1df82ac05975876fa887abc74fe700d7"
server_key_iv: "2eb6fe5a2ddd1c8cf1e7617eea9b285b"
server_key_version: "encryptionkey@0"
I20260807 07:04:52.465121 9366 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260807 07:04:52.468411 9383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:04:52.469640 9366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 07:04:52.469785 9366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "b81230f1b6424b689ebb5ca52fa568b7"
format_stamp: "Formatted at 2026-08-07 07:04:52 on dist-test-slave-ncp9"
server_key: "1df82ac05975876fa887abc74fe700d7"
server_key_iv: "2eb6fe5a2ddd1c8cf1e7617eea9b285b"
server_key_version: "encryptionkey@0"
I20260807 07:04:52.469954 9366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:04:52.484163 9366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:04:52.488237 9366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:04:52.488507 9366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:04:52.508040 9366 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.62:34697
I20260807 07:04:52.508081 9456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.62:34697 every 8 connection(s)
I20260807 07:04:52.509510 9366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:04:52.513422 9458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:04:52.516745 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 9366
I20260807 07:04:52.516987 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:04:52.517618 21836 external_mini_cluster.cc:1428] Setting key 37d200ea735fad4582ad81ed65cd2afd
I20260807 07:04:52.525971 9458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b81230f1b6424b689ebb5ca52fa568b7: Bootstrap starting.
I20260807 07:04:52.529798 9458 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b81230f1b6424b689ebb5ca52fa568b7: Neither blocks nor log segments found. Creating new log.
I20260807 07:04:52.531177 9458 log.cc:824] T 00000000000000000000000000000000 P b81230f1b6424b689ebb5ca52fa568b7: Log is configured to *not* fsync() on all Append() calls
Aug 07 07:04:52 dist-test-slave-ncp9 krb5kdc[9346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:04:52.534662 9458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b81230f1b6424b689ebb5ca52fa568b7: No bootstrap required, opened a new log
I20260807 07:04:52.540692 9462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:52.519433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59928 (local address 127.21.83.62:34697)
0807 07:04:52.520221 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:52.520232 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:52.520273 (+ 41us) server_negotiation.cc:407] Connection header received
0807 07:04:52.521983 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:52.522008 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:52.522414 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:52.522734 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:04:52.524480 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:52.525824 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:52.527615 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:52.528311 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:52.528436 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:52.533708 (+ 5272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:04:52.533717 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:04:52.533732 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:04:52.533772 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:04:52.536798 (+ 3026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:52.537613 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:52.537619 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:52.537625 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:52.537796 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:52.538158 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:52.538161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:52.538163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:52.538683 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:04:52.538889 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:52.539333 (+ 444us) server_negotiation.cc:299] Negotiation successful
0807 07:04:52.539671 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":351,"threads_started":1}
I20260807 07:04:52.543654 9458 raft_consensus.cc:348] T 00000000000000000000000000000000 P b81230f1b6424b689ebb5ca52fa568b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b81230f1b6424b689ebb5ca52fa568b7" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 34697 } }
I20260807 07:04:52.544068 9458 raft_consensus.cc:374] T 00000000000000000000000000000000 P b81230f1b6424b689ebb5ca52fa568b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:04:52.544175 9458 raft_consensus.cc:729] T 00000000000000000000000000000000 P b81230f1b6424b689ebb5ca52fa568b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b81230f1b6424b689ebb5ca52fa568b7, State: Initialized, Role: FOLLOWER
I20260807 07:04:52.544406 21836 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 07:04:52.544718 9458 consensus_queue.cc:260] T 00000000000000000000000000000000 P b81230f1b6424b689ebb5ca52fa568b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b81230f1b6424b689ebb5ca52fa568b7" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 34697 } }
I20260807 07:04:52.544960 9458 raft_consensus.cc:388] T 00000000000000000000000000000000 P b81230f1b6424b689ebb5ca52fa568b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:04:52.545095 9458 raft_consensus.cc:482] T 00000000000000000000000000000000 P b81230f1b6424b689ebb5ca52fa568b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:04:52.545252 9458 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b81230f1b6424b689ebb5ca52fa568b7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:04:52.547497 9458 raft_consensus.cc:504] T 00000000000000000000000000000000 P b81230f1b6424b689ebb5ca52fa568b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b81230f1b6424b689ebb5ca52fa568b7" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 34697 } }
I20260807 07:04:52.547921 9458 leader_election.cc:302] T 00000000000000000000000000000000 P b81230f1b6424b689ebb5ca52fa568b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b81230f1b6424b689ebb5ca52fa568b7; no voters:
I20260807 07:04:52.548439 9458 leader_election.cc:288] T 00000000000000000000000000000000 P b81230f1b6424b689ebb5ca52fa568b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 07:04:52.548549 9464 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b81230f1b6424b689ebb5ca52fa568b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:04:52.549187 9464 raft_consensus.cc:686] T 00000000000000000000000000000000 P b81230f1b6424b689ebb5ca52fa568b7 [term 1 LEADER]: Becoming Leader. State: Replica: b81230f1b6424b689ebb5ca52fa568b7, State: Running, Role: LEADER
I20260807 07:04:52.549669 9464 consensus_queue.cc:237] T 00000000000000000000000000000000 P b81230f1b6424b689ebb5ca52fa568b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b81230f1b6424b689ebb5ca52fa568b7" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 34697 } }
I20260807 07:04:52.550266 9458 sys_catalog.cc:564] T 00000000000000000000000000000000 P b81230f1b6424b689ebb5ca52fa568b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:04:52.552043 9465 sys_catalog.cc:455] T 00000000000000000000000000000000 P b81230f1b6424b689ebb5ca52fa568b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b81230f1b6424b689ebb5ca52fa568b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b81230f1b6424b689ebb5ca52fa568b7" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 34697 } } }
I20260807 07:04:52.552430 9465 sys_catalog.cc:458] T 00000000000000000000000000000000 P b81230f1b6424b689ebb5ca52fa568b7 [sys.catalog]: This master's current role is: LEADER
I20260807 07:04:52.552538 9466 sys_catalog.cc:455] T 00000000000000000000000000000000 P b81230f1b6424b689ebb5ca52fa568b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b81230f1b6424b689ebb5ca52fa568b7. Latest consensus state: current_term: 1 leader_uuid: "b81230f1b6424b689ebb5ca52fa568b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b81230f1b6424b689ebb5ca52fa568b7" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 34697 } } }
I20260807 07:04:52.552876 9466 sys_catalog.cc:458] T 00000000000000000000000000000000 P b81230f1b6424b689ebb5ca52fa568b7 [sys.catalog]: This master's current role is: LEADER
I20260807 07:04:52.553289 9474 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:04:52.558259 9474 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:04:52.566347 9474 catalog_manager.cc:1347] Generated new cluster ID: 3a35705c109c4e418848c653b1e70276
I20260807 07:04:52.566438 9474 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:04:52.580219 9474 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:04:52.581435 9474 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:04:52.594779 9474 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b81230f1b6424b689ebb5ca52fa568b7: Generated new TSK 0
I20260807 07:04:52.595453 9474 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 07:04:52.603637 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 9366
I20260807 07:04:52.604692 9448 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:04:52.694231 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 9366
2026-08-07T07:04:52Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2132 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 07:04:52 dist-test-slave-ncp9 krb5kdc[9502](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 07:04:52 dist-test-slave-ncp9 krb5kdc[9502](info): set up 2 sockets
Aug 07 07:04:52 dist-test-slave-ncp9 krb5kdc[9502](info): commencing operation
krb5kdc: starting...
W20260807 07:04:53.893251 21836 mini_kdc.cc:121] Time spent starting KDC: real 1.157s user 0.008s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Aug 07 07:04:54 dist-test-slave-ncp9 krb5kdc[9502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T07:04:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:04:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
I20260807 07:04:54.198663 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:40691
--webserver_interface=127.21.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:37735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:40691
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:04:54.526221 9519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:04:54.526585 9519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:04:54.526719 9519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:04:54.535583 9519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:04:54.535759 9519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:04:54.535846 9519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:04:54.535933 9519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:04:54.536010 9519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:04:54.547048 9519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:37735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:40691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:40691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:04:54.548431 9519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:04:54.549789 9519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:04:54.560843 9529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:04:54.560881 9527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:04:54.560974 9526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:04:54.561942 9519 server_base.cc:1034] running on GCE node
I20260807 07:04:54.562630 9519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:04:54.563925 9519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:04:54.565124 9519 hybrid_clock.cc:648] HybridClock initialized: now 1786086294565075 us; error 40 us; skew 500 ppm
I20260807 07:04:54.565480 9519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:04:54 dist-test-slave-ncp9 krb5kdc[9502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086294, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:04:54.568397 9519 init.cc:377] Logged in from keytab as kudu/127.21.83.62@KRBTEST.COM (short username kudu)
I20260807 07:04:54.569849 9519 webserver.cc:460] Webserver started at http://127.21.83.62:39817/ using document root <none> and password file <none>
I20260807 07:04:54.570475 9519 fs_manager.cc:359] Metadata directory not provided
I20260807 07:04:54.570595 9519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:04:54.570894 9519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:04:54.573833 9519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "189302c9464749e7a638be128648baad"
format_stamp: "Formatted at 2026-08-07 07:04:54 on dist-test-slave-ncp9"
server_key: "8361948d5d4132a9cd9b764014e18406"
server_key_iv: "e35ee20c7aec6fa64cede26b417bf78d"
server_key_version: "encryptionkey@0"
I20260807 07:04:54.574563 9519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "189302c9464749e7a638be128648baad"
format_stamp: "Formatted at 2026-08-07 07:04:54 on dist-test-slave-ncp9"
server_key: "8361948d5d4132a9cd9b764014e18406"
server_key_iv: "e35ee20c7aec6fa64cede26b417bf78d"
server_key_version: "encryptionkey@0"
I20260807 07:04:54.579596 9519 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260807 07:04:54.582978 9543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:04:54.584255 9519 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 07:04:54.584419 9519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "189302c9464749e7a638be128648baad"
format_stamp: "Formatted at 2026-08-07 07:04:54 on dist-test-slave-ncp9"
server_key: "8361948d5d4132a9cd9b764014e18406"
server_key_iv: "e35ee20c7aec6fa64cede26b417bf78d"
server_key_version: "encryptionkey@0"
I20260807 07:04:54.584619 9519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:04:54.618628 9519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:04:54.622488 9519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:04:54.622761 9519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:04:54.640203 9519 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.62:40691
I20260807 07:04:54.640242 9613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.62:40691 every 8 connection(s)
I20260807 07:04:54.641490 9519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:04:54.644838 9614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:04:54.648479 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 9519
I20260807 07:04:54.648723 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:04:54.649583 21836 external_mini_cluster.cc:1428] Setting key a94bbea7776b1883e7b15c6a3ecbae2c
I20260807 07:04:54.657655 9614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad: Bootstrap starting.
I20260807 07:04:54.661128 9614 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad: Neither blocks nor log segments found. Creating new log.
I20260807 07:04:54.662400 9614 log.cc:824] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad: Log is configured to *not* fsync() on all Append() calls
Aug 07 07:04:54 dist-test-slave-ncp9 krb5kdc[9502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:04:54.665311 9614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad: No bootstrap required, opened a new log
I20260807 07:04:54.672031 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:54.651546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58672 (local address 127.21.83.62:40691)
0807 07:04:54.652281 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:54.652291 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:54.652328 (+ 37us) server_negotiation.cc:407] Connection header received
0807 07:04:54.653886 (+ 1558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:54.653907 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:54.654263 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:54.654558 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:04:54.656290 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:54.657617 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:54.659455 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:54.660076 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:54.660205 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:54.665128 (+ 4923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:04:54.665135 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:04:54.665149 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:04:54.665183 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:04:54.668139 (+ 2956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:54.668976 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:54.668983 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:54.668989 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:54.669180 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:54.669550 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:54.669553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:54.669555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:54.670041 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:04:54.670284 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:54.670740 (+ 456us) server_negotiation.cc:299] Negotiation successful
0807 07:04:54.671064 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":342,"threads_started":1}
I20260807 07:04:54.673769 9614 raft_consensus.cc:348] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "189302c9464749e7a638be128648baad" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 40691 } }
I20260807 07:04:54.674096 9614 raft_consensus.cc:374] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:04:54.674213 9614 raft_consensus.cc:729] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 189302c9464749e7a638be128648baad, State: Initialized, Role: FOLLOWER
I20260807 07:04:54.674690 9614 consensus_queue.cc:260] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "189302c9464749e7a638be128648baad" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 40691 } }
I20260807 07:04:54.674902 9614 raft_consensus.cc:388] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:04:54.675021 9614 raft_consensus.cc:482] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:04:54.675165 9614 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:04:54.677242 9614 raft_consensus.cc:504] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "189302c9464749e7a638be128648baad" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 40691 } }
I20260807 07:04:54.677665 9614 leader_election.cc:302] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 189302c9464749e7a638be128648baad; no voters:
I20260807 07:04:54.678195 9614 leader_election.cc:288] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 07:04:54.678301 9619 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:04:54.679317 9619 raft_consensus.cc:686] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad [term 1 LEADER]: Becoming Leader. State: Replica: 189302c9464749e7a638be128648baad, State: Running, Role: LEADER
I20260807 07:04:54.679908 9619 consensus_queue.cc:237] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "189302c9464749e7a638be128648baad" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 40691 } }
I20260807 07:04:54.680163 9614 sys_catalog.cc:564] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:04:54.682677 9624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 189302c9464749e7a638be128648baad. Latest consensus state: current_term: 1 leader_uuid: "189302c9464749e7a638be128648baad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "189302c9464749e7a638be128648baad" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 40691 } } }
I20260807 07:04:54.683121 9624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad [sys.catalog]: This master's current role is: LEADER
I20260807 07:04:54.683059 9623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "189302c9464749e7a638be128648baad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "189302c9464749e7a638be128648baad" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 40691 } } }
I20260807 07:04:54.683393 9623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad [sys.catalog]: This master's current role is: LEADER
I20260807 07:04:54.684885 9630 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:04:54.689473 9630 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:04:54.698194 9630 catalog_manager.cc:1347] Generated new cluster ID: f014a38f54fe49d19698d571a1c22ed8
I20260807 07:04:54.698294 9630 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:04:54.750125 9630 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:04:54.751976 9630 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:04:54.760540 9630 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad: Generated new TSK 0
I20260807 07:04:54.761551 9630 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.83.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab.
Entry for principal kudu/127.21.83.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab.
I20260807 07:04:54.835852 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.1:0
--local_ip_for_outbound_sockets=127.21.83.1
--webserver_interface=127.21.83.1
--webserver_port=0
--tserver_master_addrs=127.21.83.62:40691
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab
--principal=kudu/127.21.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:37735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:04:55.180235 9650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:04:55.180483 9650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:04:55.180581 9650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:04:55.187940 9650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:04:55.188036 9650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:04:55.188180 9650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.1
I20260807 07:04:55.197284 9650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:37735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab
--principal=kudu/127.21.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.83.1
--webserver_port=0
--tserver_master_addrs=127.21.83.62:40691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.1
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:04:55.198347 9650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:04:55.199467 9650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:04:55.211510 9662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:04:55.212078 9656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:04:55.212090 9657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:04:55.212667 9650 server_base.cc:1034] running on GCE node
I20260807 07:04:55.213406 9650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:04:55.214591 9650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:04:55.215780 9650 hybrid_clock.cc:648] HybridClock initialized: now 1786086295215727 us; error 51 us; skew 500 ppm
I20260807 07:04:55.216073 9650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:04:55 dist-test-slave-ncp9 krb5kdc[9502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086295, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:04:55.224421 9650 init.cc:377] Logged in from keytab as kudu/127.21.83.1@KRBTEST.COM (short username kudu)
I20260807 07:04:55.226030 9650 webserver.cc:460] Webserver started at http://127.21.83.1:38133/ using document root <none> and password file <none>
I20260807 07:04:55.226724 9650 fs_manager.cc:359] Metadata directory not provided
I20260807 07:04:55.226830 9650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:04:55.227077 9650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:04:55.229979 9650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-0/data/instance:
uuid: "7676cbdb92e040f9b34aef99c0ddbc30"
format_stamp: "Formatted at 2026-08-07 07:04:55 on dist-test-slave-ncp9"
server_key: "c2205ded58806d934c6bd252549406ab"
server_key_iv: "d5f9ea3655889aff94521f9360313c0d"
server_key_version: "encryptionkey@0"
I20260807 07:04:55.230715 9650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-0/wal/instance:
uuid: "7676cbdb92e040f9b34aef99c0ddbc30"
format_stamp: "Formatted at 2026-08-07 07:04:55 on dist-test-slave-ncp9"
server_key: "c2205ded58806d934c6bd252549406ab"
server_key_iv: "d5f9ea3655889aff94521f9360313c0d"
server_key_version: "encryptionkey@0"
I20260807 07:04:55.235507 9650 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260807 07:04:55.238710 9677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:04:55.239863 9650 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260807 07:04:55.239995 9650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-0/wal
uuid: "7676cbdb92e040f9b34aef99c0ddbc30"
format_stamp: "Formatted at 2026-08-07 07:04:55 on dist-test-slave-ncp9"
server_key: "c2205ded58806d934c6bd252549406ab"
server_key_iv: "d5f9ea3655889aff94521f9360313c0d"
server_key_version: "encryptionkey@0"
I20260807 07:04:55.240165 9650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:04:55.262317 9650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:04:55.267223 9650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:04:55.267481 9650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:04:55.268622 9650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:04:55.270346 9650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:04:55.270447 9650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:04:55.270642 9650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:04:55.270725 9650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:04:55.314548 9650 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.1:45401
I20260807 07:04:55.314594 9822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.1:45401 every 8 connection(s)
I20260807 07:04:55.315732 9650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
I20260807 07:04:55.320137 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 9650
I20260807 07:04:55.320367 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-0/wal/instance
I20260807 07:04:55.321189 21836 external_mini_cluster.cc:1428] Setting key e80a77c772aa47b96641f8787ebe2c81
Aug 07 07:04:55 dist-test-slave-ncp9 krb5kdc[9502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086295, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.1@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:04:55.336001 9827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:55.318131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:54683 (local address 127.21.83.62:40691)
0807 07:04:55.318560 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:55.318564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:55.319482 (+ 918us) server_negotiation.cc:407] Connection header received
0807 07:04:55.322070 (+ 2588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:55.322076 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:55.322153 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:55.322261 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:04:55.323920 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:55.324643 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:55.326702 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:55.327091 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:55.327184 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:55.330755 (+ 3571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:04:55.330760 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:04:55.330764 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:04:55.330795 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:04:55.333206 (+ 2411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:55.333859 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:55.333864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:55.333866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:55.333979 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:55.334383 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:55.334387 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:55.334389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:55.334634 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:04:55.334764 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:55.335447 (+ 683us) server_negotiation.cc:299] Negotiation successful
0807 07:04:55.335654 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":215,"threads_started":1}
I20260807 07:04:55.336779 9828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:55.318642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.1:54683)
0807 07:04:55.319332 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:55.319369 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0807 07:04:55.321707 (+ 2338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:04:55.322498 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:04:55.322507 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:04:55.322985 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:04:55.323731 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:55.323759 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:55.324810 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:55.324825 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:55.326577 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:55.326592 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:55.327297 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:55.327303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:55.327544 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:55.328195 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:04:55.328222 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:04:55.330556 (+ 2334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:04:55.333330 (+ 2774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:04:55.333339 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:04:55.333352 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:04:55.333740 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:04:55.334079 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:04:55.334092 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:04:55.334096 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:04:55.334271 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:04:55.334754 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:04:55.334761 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:04:55.335044 (+ 283us) client_negotiation.cc:769] Sending connection context
0807 07:04:55.335297 (+ 253us) client_negotiation.cc:240] Negotiation successful
0807 07:04:55.335714 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":263,"threads_started":1}
I20260807 07:04:55.338287 9825 heartbeater.cc:347] Connected to a master server at 127.21.83.62:40691
I20260807 07:04:55.338552 9825 heartbeater.cc:464] Registering TS with master...
I20260807 07:04:55.339197 9825 heartbeater.cc:511] Master 127.21.83.62:40691 requested a full tablet report, sending...
I20260807 07:04:55.341413 9568 ts_manager.cc:194] Registered new tserver with Master: 7676cbdb92e040f9b34aef99c0ddbc30 (127.21.83.1:45401)
I20260807 07:04:55.343343 9568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.1@KRBTEST.COM'} at 127.21.83.1:54683
WARNING: no policy specified for kudu/127.21.83.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab.
Entry for principal kudu/127.21.83.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab.
I20260807 07:04:55.386229 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.2:0
--local_ip_for_outbound_sockets=127.21.83.2
--webserver_interface=127.21.83.2
--webserver_port=0
--tserver_master_addrs=127.21.83.62:40691
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab
--principal=kudu/127.21.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:37735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:04:55.720353 9832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:04:55.720666 9832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:04:55.720818 9832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:04:55.728539 9832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:04:55.728684 9832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:04:55.728854 9832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.2
I20260807 07:04:55.738610 9832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:37735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab
--principal=kudu/127.21.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.83.2
--webserver_port=0
--tserver_master_addrs=127.21.83.62:40691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.2
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:04:55.739794 9832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:04:55.741086 9832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:04:55.753531 9843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:04:55.753600 9848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:04:55.753506 9846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:04:55.754428 9832 server_base.cc:1034] running on GCE node
I20260807 07:04:55.754940 9832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:04:55.755740 9832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:04:55.756944 9832 hybrid_clock.cc:648] HybridClock initialized: now 1786086295756890 us; error 42 us; skew 500 ppm
I20260807 07:04:55.757227 9832 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:04:55 dist-test-slave-ncp9 krb5kdc[9502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086295, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:04:55.760154 9832 init.cc:377] Logged in from keytab as kudu/127.21.83.2@KRBTEST.COM (short username kudu)
I20260807 07:04:55.761586 9832 webserver.cc:460] Webserver started at http://127.21.83.2:36915/ using document root <none> and password file <none>
I20260807 07:04:55.762223 9832 fs_manager.cc:359] Metadata directory not provided
I20260807 07:04:55.762341 9832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:04:55.762622 9832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:04:55.765486 9832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-1/data/instance:
uuid: "fe73acb9913948e09ae60afc1ebdf24f"
format_stamp: "Formatted at 2026-08-07 07:04:55 on dist-test-slave-ncp9"
server_key: "10d60cbdc41369e1502319772e11b105"
server_key_iv: "a99087ea8866ef68b76a9cf1039ce550"
server_key_version: "encryptionkey@0"
I20260807 07:04:55.766228 9832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-1/wal/instance:
uuid: "fe73acb9913948e09ae60afc1ebdf24f"
format_stamp: "Formatted at 2026-08-07 07:04:55 on dist-test-slave-ncp9"
server_key: "10d60cbdc41369e1502319772e11b105"
server_key_iv: "a99087ea8866ef68b76a9cf1039ce550"
server_key_version: "encryptionkey@0"
I20260807 07:04:55.771117 9832 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 07:04:55.774452 9856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:04:55.775662 9832 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260807 07:04:55.775804 9832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-1/wal
uuid: "fe73acb9913948e09ae60afc1ebdf24f"
format_stamp: "Formatted at 2026-08-07 07:04:55 on dist-test-slave-ncp9"
server_key: "10d60cbdc41369e1502319772e11b105"
server_key_iv: "a99087ea8866ef68b76a9cf1039ce550"
server_key_version: "encryptionkey@0"
I20260807 07:04:55.775990 9832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:04:55.793649 9832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:04:55.797107 9832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:04:55.797354 9832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:04:55.798205 9832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:04:55.799419 9832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:04:55.799495 9832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:04:55.799619 9832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:04:55.799674 9832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:04:55.832100 9832 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.2:45093
I20260807 07:04:55.832157 10006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.2:45093 every 8 connection(s)
I20260807 07:04:55.833491 9832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
I20260807 07:04:55.841957 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 9832
I20260807 07:04:55.842147 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-1/wal/instance
I20260807 07:04:55.842618 21836 external_mini_cluster.cc:1428] Setting key 3afc2697ee3943cb7a09335d043b9b2f
Aug 07 07:04:55 dist-test-slave-ncp9 krb5kdc[9502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086295, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.2@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:04:55.858470 9827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:55.836139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:60905 (local address 127.21.83.62:40691)
0807 07:04:55.836348 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:55.836353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:55.837647 (+ 1294us) server_negotiation.cc:407] Connection header received
0807 07:04:55.839568 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:55.839575 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:55.839668 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:55.839769 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:04:55.841887 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:55.842759 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:55.845067 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:55.845526 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:55.845635 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:55.850877 (+ 5242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:04:55.850883 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:04:55.850887 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:04:55.850938 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:04:55.853867 (+ 2929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:55.854974 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:55.854979 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:55.854982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:55.855148 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:55.855832 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:55.855840 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:55.855843 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:55.856353 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:04:55.856542 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:55.857689 (+ 1147us) server_negotiation.cc:299] Negotiation successful
0807 07:04:55.858026 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 07:04:55.859560 10009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:55.836666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.2:60905)
0807 07:04:55.837481 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:55.837520 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0807 07:04:55.839253 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:04:55.840042 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:04:55.840052 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:04:55.840663 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:04:55.841531 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:55.841559 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:55.843003 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:55.843007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:55.844887 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:55.844935 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:55.845814 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:55.845823 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:55.846136 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:55.847032 (+ 896us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:04:55.847072 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:04:55.850503 (+ 3431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:04:55.854082 (+ 3579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:04:55.854093 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:04:55.854114 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:04:55.854789 (+ 675us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:04:55.855268 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:04:55.855273 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:04:55.855276 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:04:55.855543 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:04:55.856537 (+ 994us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:04:55.856546 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:04:55.857117 (+ 571us) client_negotiation.cc:769] Sending connection context
0807 07:04:55.857470 (+ 353us) client_negotiation.cc:240] Negotiation successful
0807 07:04:55.858096 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":333,"threads_started":1}
I20260807 07:04:55.861438 10007 heartbeater.cc:347] Connected to a master server at 127.21.83.62:40691
I20260807 07:04:55.861778 10007 heartbeater.cc:464] Registering TS with master...
I20260807 07:04:55.862607 10007 heartbeater.cc:511] Master 127.21.83.62:40691 requested a full tablet report, sending...
I20260807 07:04:55.864492 9568 ts_manager.cc:194] Registered new tserver with Master: fe73acb9913948e09ae60afc1ebdf24f (127.21.83.2:45093)
I20260807 07:04:55.865409 9568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.2@KRBTEST.COM'} at 127.21.83.2:60905
WARNING: no policy specified for kudu/127.21.83.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab.
Entry for principal kudu/127.21.83.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab.
I20260807 07:04:55.912370 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.3:0
--local_ip_for_outbound_sockets=127.21.83.3
--webserver_interface=127.21.83.3
--webserver_port=0
--tserver_master_addrs=127.21.83.62:40691
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab
--principal=kudu/127.21.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:37735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:04:56.265993 10012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:04:56.266302 10012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:04:56.266439 10012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:04:56.273840 10012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:04:56.273965 10012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:04:56.274129 10012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.3
I20260807 07:04:56.284056 10012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:37735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab
--principal=kudu/127.21.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.83.3
--webserver_port=0
--tserver_master_addrs=127.21.83.62:40691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.3
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:04:56.285400 10012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:04:56.286695 10012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:04:56.299391 10019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:04:56.299547 10022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:04:56.299540 10020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:04:56.300422 10012 server_base.cc:1034] running on GCE node
I20260807 07:04:56.301051 10012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:04:56.301952 10012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:04:56.303177 10012 hybrid_clock.cc:648] HybridClock initialized: now 1786086296303101 us; error 64 us; skew 500 ppm
I20260807 07:04:56.303491 10012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:04:56 dist-test-slave-ncp9 krb5kdc[9502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086296, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:04:56.306553 10012 init.cc:377] Logged in from keytab as kudu/127.21.83.3@KRBTEST.COM (short username kudu)
I20260807 07:04:56.307945 10012 webserver.cc:460] Webserver started at http://127.21.83.3:33465/ using document root <none> and password file <none>
I20260807 07:04:56.308583 10012 fs_manager.cc:359] Metadata directory not provided
I20260807 07:04:56.308702 10012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:04:56.309032 10012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:04:56.311991 10012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-2/data/instance:
uuid: "7882063db22d46a39672797ac9638ce4"
format_stamp: "Formatted at 2026-08-07 07:04:56 on dist-test-slave-ncp9"
server_key: "9a7eedc45af41856860fbb6aef7e54ba"
server_key_iv: "bb8699a76a13c87918cb06bbfc46f5f9"
server_key_version: "encryptionkey@0"
I20260807 07:04:56.313544 10012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-2/wal/instance:
uuid: "7882063db22d46a39672797ac9638ce4"
format_stamp: "Formatted at 2026-08-07 07:04:56 on dist-test-slave-ncp9"
server_key: "9a7eedc45af41856860fbb6aef7e54ba"
server_key_iv: "bb8699a76a13c87918cb06bbfc46f5f9"
server_key_version: "encryptionkey@0"
I20260807 07:04:56.319324 10012 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 07:04:56.323155 10030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:04:56.324694 10012 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 07:04:56.324872 10012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-2/wal
uuid: "7882063db22d46a39672797ac9638ce4"
format_stamp: "Formatted at 2026-08-07 07:04:56 on dist-test-slave-ncp9"
server_key: "9a7eedc45af41856860fbb6aef7e54ba"
server_key_iv: "bb8699a76a13c87918cb06bbfc46f5f9"
server_key_version: "encryptionkey@0"
I20260807 07:04:56.325152 10012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:04:56.337090 10012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:04:56.340632 10012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:04:56.340879 10012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:04:56.341809 10012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:04:56.343118 10012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:04:56.343191 10012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:04:56.343304 10012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:04:56.343353 10012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:04:56.346235 9825 heartbeater.cc:503] Master 127.21.83.62:40691 was elected leader, sending a full tablet report...
I20260807 07:04:56.373450 10012 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.3:45141
I20260807 07:04:56.373495 10192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.3:45141 every 8 connection(s)
I20260807 07:04:56.374714 10012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
I20260807 07:04:56.384526 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 10012
I20260807 07:04:56.384712 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-2/wal/instance
I20260807 07:04:56.385326 21836 external_mini_cluster.cc:1428] Setting key b054c7ee70de327cac259140c5547e90
Aug 07 07:04:56 dist-test-slave-ncp9 krb5kdc[9502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086296, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.3@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:04:56.394658 10196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:56.377136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:52809 (local address 127.21.83.62:40691)
0807 07:04:56.377646 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:56.377652 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:56.378600 (+ 948us) server_negotiation.cc:407] Connection header received
0807 07:04:56.380621 (+ 2021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:56.380626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:56.380696 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:56.380798 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:04:56.382338 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:56.383150 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:56.384725 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:56.385330 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:56.385461 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:56.389134 (+ 3673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:04:56.389139 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:04:56.389142 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:04:56.389177 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:04:56.391741 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:56.392423 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:56.392427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:56.392429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:56.392561 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:56.392982 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:56.392986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:56.392988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:56.393251 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:04:56.393388 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:56.394058 (+ 670us) server_negotiation.cc:299] Negotiation successful
0807 07:04:56.394285 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":223,"threads_started":1}
I20260807 07:04:56.395478 10197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:56.377638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.3:52809)
0807 07:04:56.378422 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:56.378469 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 07:04:56.380319 (+ 1850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:04:56.381027 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:04:56.381038 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:04:56.381578 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:04:56.382156 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:56.382173 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:56.383316 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:56.383320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:56.384555 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:56.384573 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:56.385539 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:56.385544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:56.385739 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:56.386387 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:04:56.386409 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:04:56.388937 (+ 2528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:04:56.391869 (+ 2932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:04:56.391878 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:04:56.391893 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:04:56.392304 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:04:56.392661 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:04:56.392666 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:04:56.392670 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:04:56.392857 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:04:56.393374 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:04:56.393382 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:04:56.393695 (+ 313us) client_negotiation.cc:769] Sending connection context
0807 07:04:56.393920 (+ 225us) client_negotiation.cc:240] Negotiation successful
0807 07:04:56.394423 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":280,"threads_started":1}
I20260807 07:04:56.396708 10193 heartbeater.cc:347] Connected to a master server at 127.21.83.62:40691
I20260807 07:04:56.397008 10193 heartbeater.cc:464] Registering TS with master...
I20260807 07:04:56.397665 10193 heartbeater.cc:511] Master 127.21.83.62:40691 requested a full tablet report, sending...
I20260807 07:04:56.398983 9568 ts_manager.cc:194] Registered new tserver with Master: 7882063db22d46a39672797ac9638ce4 (127.21.83.3:45141)
I20260807 07:04:56.399890 9568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.3@KRBTEST.COM'} at 127.21.83.3:52809
I20260807 07:04:56.400632 21836 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 07:04:56.416687 10196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:56.405101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58678 (local address 127.21.83.62:40691)
0807 07:04:56.405248 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:56.405253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:56.405666 (+ 413us) server_negotiation.cc:407] Connection header received
0807 07:04:56.405785 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:56.405788 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:56.405852 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:56.405925 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:04:56.407234 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:56.407924 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:56.409174 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:56.409551 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:56.409639 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:56.411751 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:04:56.411756 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:04:56.411760 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:04:56.411790 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:04:56.414271 (+ 2481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:56.415005 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:56.415009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:56.415011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:56.415137 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:56.415428 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:56.415431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:56.415433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:56.415757 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:04:56.415949 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:56.416274 (+ 325us) server_negotiation.cc:299] Negotiation successful
0807 07:04:56.416454 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 07:04:56.421111 9567 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58678:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 07:04:56.428633 9567 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 07:04:56.462386 10218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:56.444083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52876 (local address 127.21.83.1:45401)
0807 07:04:56.452609 (+ 8526us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:56.452620 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:56.453747 (+ 1127us) server_negotiation.cc:407] Connection header received
0807 07:04:56.453883 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:56.453893 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:56.454175 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:56.455165 (+ 990us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:04:56.455415 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:56.457007 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:56.459821 (+ 2814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:56.460987 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:56.461109 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:56.461344 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:56.461952 (+ 608us) server_negotiation.cc:299] Negotiation successful
0807 07:04:56.462122 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8352,"thread_start_us":8160,"threads_started":1}
I20260807 07:04:56.462440 10212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:56.442029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.2:45093 (local address 127.0.0.1:43306)
0807 07:04:56.443088 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:56.443142 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0807 07:04:56.443331 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:04:56.455201 (+ 11870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:04:56.455206 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:04:56.455231 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:04:56.455726 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:56.455740 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:56.457578 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:56.457583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:56.459782 (+ 2199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:56.459793 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:56.461368 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:56.461374 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:56.461605 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:56.461670 (+ 65us) client_negotiation.cc:769] Sending connection context
0807 07:04:56.461777 (+ 107us) client_negotiation.cc:240] Negotiation successful
0807 07:04:56.461973 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"mutex_wait_us":24,"thread_start_us":643,"threads_started":1}
I20260807 07:04:56.463050 10216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:56.443564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.3:45141 (local address 127.0.0.1:42432)
0807 07:04:56.455885 (+ 12321us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:56.455901 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 07:04:56.456007 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:04:56.456458 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:04:56.456463 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:04:56.456485 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:04:56.456842 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:56.456855 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:56.458525 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:56.458530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:56.460201 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:56.460216 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:56.462183 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:56.462207 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:56.462374 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:56.462408 (+ 34us) client_negotiation.cc:769] Sending connection context
0807 07:04:56.462499 (+ 91us) client_negotiation.cc:240] Negotiation successful
0807 07:04:56.462625 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12201,"mutex_wait_us":25,"thread_start_us":12004,"threads_started":1}
I20260807 07:04:56.463095 10215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:56.443130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:45401 (local address 127.0.0.1:52876)
0807 07:04:56.453415 (+ 10285us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:56.453435 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 07:04:56.453606 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:04:56.454437 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:04:56.454445 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:04:56.454479 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:04:56.454932 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:56.454944 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:56.457158 (+ 2214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:56.457163 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:56.459648 (+ 2485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:56.459663 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:56.461503 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:56.461509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:56.461704 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:56.461738 (+ 34us) client_negotiation.cc:769] Sending connection context
0807 07:04:56.461821 (+ 83us) client_negotiation.cc:240] Negotiation successful
0807 07:04:56.462679 (+ 858us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10158,"mutex_wait_us":710,"thread_start_us":10021,"threads_started":1}
I20260807 07:04:56.463349 10214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:56.442289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43306 (local address 127.21.83.2:45093)
0807 07:04:56.454762 (+ 12473us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:56.454768 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:56.454790 (+ 22us) server_negotiation.cc:407] Connection header received
0807 07:04:56.454863 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:56.454867 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:56.455012 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:56.455187 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:04:56.455936 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:56.457417 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:56.459926 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:56.461149 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:56.462669 (+ 1520us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:56.462848 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:56.462915 (+ 67us) server_negotiation.cc:299] Negotiation successful
0807 07:04:56.463077 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12331,"thread_start_us":12223,"threads_started":1}
I20260807 07:04:56.464504 10219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:56.444474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42432 (local address 127.21.83.3:45141)
0807 07:04:56.453391 (+ 8917us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:56.453399 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:56.455994 (+ 2595us) server_negotiation.cc:407] Connection header received
0807 07:04:56.456146 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:56.456152 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:56.456297 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:56.456447 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:04:56.457058 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:56.458387 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:56.460373 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:56.461713 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:56.461823 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:56.462025 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:56.463346 (+ 1321us) server_negotiation.cc:299] Negotiation successful
0807 07:04:56.463511 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8762,"thread_start_us":8308,"threads_started":1}
I20260807 07:04:56.466817 9917 tablet_service.cc:1467] Processing CreateTablet for tablet 21621e14f640407ab71cb3e55c76dc36 (DEFAULT_TABLE table=test-table [id=3a7f26eedfbe49da952fa5ad0352fb0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:04:56.466814 9740 tablet_service.cc:1467] Processing CreateTablet for tablet 21621e14f640407ab71cb3e55c76dc36 (DEFAULT_TABLE table=test-table [id=3a7f26eedfbe49da952fa5ad0352fb0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:04:56.468009 9917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21621e14f640407ab71cb3e55c76dc36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:04:56.467631 10098 tablet_service.cc:1467] Processing CreateTablet for tablet 21621e14f640407ab71cb3e55c76dc36 (DEFAULT_TABLE table=test-table [id=3a7f26eedfbe49da952fa5ad0352fb0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:04:56.468211 9740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21621e14f640407ab71cb3e55c76dc36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:04:56.468953 10098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21621e14f640407ab71cb3e55c76dc36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:04:56.479202 10221 tablet_bootstrap.cc:492] T 21621e14f640407ab71cb3e55c76dc36 P fe73acb9913948e09ae60afc1ebdf24f: Bootstrap starting.
I20260807 07:04:56.482635 10221 tablet_bootstrap.cc:654] T 21621e14f640407ab71cb3e55c76dc36 P fe73acb9913948e09ae60afc1ebdf24f: Neither blocks nor log segments found. Creating new log.
I20260807 07:04:56.482949 10222 tablet_bootstrap.cc:492] T 21621e14f640407ab71cb3e55c76dc36 P 7676cbdb92e040f9b34aef99c0ddbc30: Bootstrap starting.
I20260807 07:04:56.484118 10221 log.cc:824] T 21621e14f640407ab71cb3e55c76dc36 P fe73acb9913948e09ae60afc1ebdf24f: Log is configured to *not* fsync() on all Append() calls
I20260807 07:04:56.484544 10224 tablet_bootstrap.cc:492] T 21621e14f640407ab71cb3e55c76dc36 P 7882063db22d46a39672797ac9638ce4: Bootstrap starting.
I20260807 07:04:56.486433 10222 tablet_bootstrap.cc:654] T 21621e14f640407ab71cb3e55c76dc36 P 7676cbdb92e040f9b34aef99c0ddbc30: Neither blocks nor log segments found. Creating new log.
I20260807 07:04:56.487603 10224 tablet_bootstrap.cc:654] T 21621e14f640407ab71cb3e55c76dc36 P 7882063db22d46a39672797ac9638ce4: Neither blocks nor log segments found. Creating new log.
I20260807 07:04:56.488128 10221 tablet_bootstrap.cc:492] T 21621e14f640407ab71cb3e55c76dc36 P fe73acb9913948e09ae60afc1ebdf24f: No bootstrap required, opened a new log
I20260807 07:04:56.488137 10222 log.cc:824] T 21621e14f640407ab71cb3e55c76dc36 P 7676cbdb92e040f9b34aef99c0ddbc30: Log is configured to *not* fsync() on all Append() calls
I20260807 07:04:56.488512 10221 ts_tablet_manager.cc:1397] T 21621e14f640407ab71cb3e55c76dc36 P fe73acb9913948e09ae60afc1ebdf24f: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260807 07:04:56.488801 10224 log.cc:824] T 21621e14f640407ab71cb3e55c76dc36 P 7882063db22d46a39672797ac9638ce4: Log is configured to *not* fsync() on all Append() calls
I20260807 07:04:56.491983 10222 tablet_bootstrap.cc:492] T 21621e14f640407ab71cb3e55c76dc36 P 7676cbdb92e040f9b34aef99c0ddbc30: No bootstrap required, opened a new log
I20260807 07:04:56.492270 10222 ts_tablet_manager.cc:1397] T 21621e14f640407ab71cb3e55c76dc36 P 7676cbdb92e040f9b34aef99c0ddbc30: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260807 07:04:56.495525 10224 tablet_bootstrap.cc:492] T 21621e14f640407ab71cb3e55c76dc36 P 7882063db22d46a39672797ac9638ce4: No bootstrap required, opened a new log
I20260807 07:04:56.495780 10224 ts_tablet_manager.cc:1397] T 21621e14f640407ab71cb3e55c76dc36 P 7882063db22d46a39672797ac9638ce4: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260807 07:04:56.498699 10221 raft_consensus.cc:348] T 21621e14f640407ab71cb3e55c76dc36 P fe73acb9913948e09ae60afc1ebdf24f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe73acb9913948e09ae60afc1ebdf24f" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 45093 } } peers { permanent_uuid: "7882063db22d46a39672797ac9638ce4" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 45141 } } peers { permanent_uuid: "7676cbdb92e040f9b34aef99c0ddbc30" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 45401 } }
I20260807 07:04:56.499337 10221 raft_consensus.cc:374] T 21621e14f640407ab71cb3e55c76dc36 P fe73acb9913948e09ae60afc1ebdf24f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:04:56.499461 10221 raft_consensus.cc:729] T 21621e14f640407ab71cb3e55c76dc36 P fe73acb9913948e09ae60afc1ebdf24f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe73acb9913948e09ae60afc1ebdf24f, State: Initialized, Role: FOLLOWER
I20260807 07:04:56.500222 10221 consensus_queue.cc:260] T 21621e14f640407ab71cb3e55c76dc36 P fe73acb9913948e09ae60afc1ebdf24f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe73acb9913948e09ae60afc1ebdf24f" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 45093 } } peers { permanent_uuid: "7882063db22d46a39672797ac9638ce4" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 45141 } } peers { permanent_uuid: "7676cbdb92e040f9b34aef99c0ddbc30" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 45401 } }
I20260807 07:04:56.501454 10007 heartbeater.cc:503] Master 127.21.83.62:40691 was elected leader, sending a full tablet report...
I20260807 07:04:56.502177 10221 ts_tablet_manager.cc:1428] T 21621e14f640407ab71cb3e55c76dc36 P fe73acb9913948e09ae60afc1ebdf24f: Time spent starting tablet: real 0.014s user 0.007s sys 0.005s
I20260807 07:04:56.503023 10222 raft_consensus.cc:348] T 21621e14f640407ab71cb3e55c76dc36 P 7676cbdb92e040f9b34aef99c0ddbc30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe73acb9913948e09ae60afc1ebdf24f" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 45093 } } peers { permanent_uuid: "7882063db22d46a39672797ac9638ce4" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 45141 } } peers { permanent_uuid: "7676cbdb92e040f9b34aef99c0ddbc30" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 45401 } }
I20260807 07:04:56.503465 10222 raft_consensus.cc:374] T 21621e14f640407ab71cb3e55c76dc36 P 7676cbdb92e040f9b34aef99c0ddbc30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:04:56.503590 10222 raft_consensus.cc:729] T 21621e14f640407ab71cb3e55c76dc36 P 7676cbdb92e040f9b34aef99c0ddbc30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7676cbdb92e040f9b34aef99c0ddbc30, State: Initialized, Role: FOLLOWER
I20260807 07:04:56.504173 10222 consensus_queue.cc:260] T 21621e14f640407ab71cb3e55c76dc36 P 7676cbdb92e040f9b34aef99c0ddbc30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe73acb9913948e09ae60afc1ebdf24f" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 45093 } } peers { permanent_uuid: "7882063db22d46a39672797ac9638ce4" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 45141 } } peers { permanent_uuid: "7676cbdb92e040f9b34aef99c0ddbc30" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 45401 } }
I20260807 07:04:56.506011 10222 ts_tablet_manager.cc:1428] T 21621e14f640407ab71cb3e55c76dc36 P 7676cbdb92e040f9b34aef99c0ddbc30: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
I20260807 07:04:56.505889 10224 raft_consensus.cc:348] T 21621e14f640407ab71cb3e55c76dc36 P 7882063db22d46a39672797ac9638ce4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe73acb9913948e09ae60afc1ebdf24f" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 45093 } } peers { permanent_uuid: "7882063db22d46a39672797ac9638ce4" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 45141 } } peers { permanent_uuid: "7676cbdb92e040f9b34aef99c0ddbc30" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 45401 } }
I20260807 07:04:56.506299 10224 raft_consensus.cc:374] T 21621e14f640407ab71cb3e55c76dc36 P 7882063db22d46a39672797ac9638ce4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:04:56.506392 10224 raft_consensus.cc:729] T 21621e14f640407ab71cb3e55c76dc36 P 7882063db22d46a39672797ac9638ce4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7882063db22d46a39672797ac9638ce4, State: Initialized, Role: FOLLOWER
I20260807 07:04:56.507928 10224 consensus_queue.cc:260] T 21621e14f640407ab71cb3e55c76dc36 P 7882063db22d46a39672797ac9638ce4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe73acb9913948e09ae60afc1ebdf24f" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 45093 } } peers { permanent_uuid: "7882063db22d46a39672797ac9638ce4" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 45141 } } peers { permanent_uuid: "7676cbdb92e040f9b34aef99c0ddbc30" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 45401 } }
I20260807 07:04:56.509176 10193 heartbeater.cc:503] Master 127.21.83.62:40691 was elected leader, sending a full tablet report...
I20260807 07:04:56.510987 10224 ts_tablet_manager.cc:1428] T 21621e14f640407ab71cb3e55c76dc36 P 7882063db22d46a39672797ac9638ce4: Time spent starting tablet: real 0.015s user 0.012s sys 0.001s
W20260807 07:04:56.568075 9826 tablet.cc:2369] T 21621e14f640407ab71cb3e55c76dc36 P 7676cbdb92e040f9b34aef99c0ddbc30: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 07:04:56.585595 10008 tablet.cc:2369] T 21621e14f640407ab71cb3e55c76dc36 P fe73acb9913948e09ae60afc1ebdf24f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 07:04:56.626312 10195 tablet.cc:2369] T 21621e14f640407ab71cb3e55c76dc36 P 7882063db22d46a39672797ac9638ce4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 07:04:56.822247 10232 raft_consensus.cc:482] T 21621e14f640407ab71cb3e55c76dc36 P 7676cbdb92e040f9b34aef99c0ddbc30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:04:56.822642 10232 raft_consensus.cc:504] T 21621e14f640407ab71cb3e55c76dc36 P 7676cbdb92e040f9b34aef99c0ddbc30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe73acb9913948e09ae60afc1ebdf24f" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 45093 } } peers { permanent_uuid: "7882063db22d46a39672797ac9638ce4" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 45141 } } peers { permanent_uuid: "7676cbdb92e040f9b34aef99c0ddbc30" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 45401 } }
I20260807 07:04:56.824416 10232 leader_election.cc:288] T 21621e14f640407ab71cb3e55c76dc36 P 7676cbdb92e040f9b34aef99c0ddbc30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe73acb9913948e09ae60afc1ebdf24f (127.21.83.2:45093), 7882063db22d46a39672797ac9638ce4 (127.21.83.3:45141)
I20260807 07:04:56.831985 10214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:56.825000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:34483 (local address 127.21.83.2:45093)
0807 07:04:56.825278 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:56.825285 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:56.825763 (+ 478us) server_negotiation.cc:407] Connection header received
0807 07:04:56.826053 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:56.826060 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:56.826145 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:56.826214 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:04:56.827010 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:56.827848 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:56.830114 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:56.830956 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:56.831079 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:56.831190 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:56.831555 (+ 365us) server_negotiation.cc:299] Negotiation successful
0807 07:04:56.831711 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260807 07:04:56.832051 10234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:56.824842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.2:45093 (local address 127.21.83.1:34483)
0807 07:04:56.825615 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:56.825637 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 07:04:56.825841 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:04:56.826410 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:04:56.826415 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:04:56.826443 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:04:56.826843 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:56.826858 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:56.828011 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:56.828017 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:56.829941 (+ 1924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:56.829956 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:56.831213 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:56.831220 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:56.831384 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:56.831411 (+ 27us) client_negotiation.cc:769] Sending connection context
0807 07:04:56.831497 (+ 86us) client_negotiation.cc:240] Negotiation successful
0807 07:04:56.831658 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"mutex_wait_us":22,"thread_start_us":398,"threads_started":1}
I20260807 07:04:56.832042 10219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:56.825999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:40869 (local address 127.21.83.3:45141)
0807 07:04:56.826208 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:56.826213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:56.826440 (+ 227us) server_negotiation.cc:407] Connection header received
0807 07:04:56.826615 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:56.826620 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:56.826698 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:56.826801 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:04:56.827374 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:56.828179 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:56.830176 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:56.830983 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:56.831121 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:56.831250 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:56.831654 (+ 404us) server_negotiation.cc:299] Negotiation successful
0807 07:04:56.831800 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260807 07:04:56.832801 10235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:56.825705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.3:45141 (local address 127.21.83.1:40869)
0807 07:04:56.826341 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:56.826356 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 07:04:56.826485 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:04:56.826822 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:04:56.826825 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:04:56.826844 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:04:56.827217 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:56.827230 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:56.828311 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:56.828315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:56.830020 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:56.830037 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:56.831265 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:56.831273 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:56.831443 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:56.831477 (+ 34us) client_negotiation.cc:769] Sending connection context
0807 07:04:56.832253 (+ 776us) client_negotiation.cc:240] Negotiation successful
0807 07:04:56.832403 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"mutex_wait_us":22,"spinlock_wait_cycles":19712,"thread_start_us":297,"threads_started":1}
I20260807 07:04:56.833709 10127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21621e14f640407ab71cb3e55c76dc36" candidate_uuid: "7676cbdb92e040f9b34aef99c0ddbc30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7882063db22d46a39672797ac9638ce4" is_pre_election: true
I20260807 07:04:56.833709 9948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21621e14f640407ab71cb3e55c76dc36" candidate_uuid: "7676cbdb92e040f9b34aef99c0ddbc30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe73acb9913948e09ae60afc1ebdf24f" is_pre_election: true
I20260807 07:04:56.834232 10127 raft_consensus.cc:2454] T 21621e14f640407ab71cb3e55c76dc36 P 7882063db22d46a39672797ac9638ce4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7676cbdb92e040f9b34aef99c0ddbc30 in term 0.
I20260807 07:04:56.834323 9948 raft_consensus.cc:2454] T 21621e14f640407ab71cb3e55c76dc36 P fe73acb9913948e09ae60afc1ebdf24f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7676cbdb92e040f9b34aef99c0ddbc30 in term 0.
I20260807 07:04:56.834918 9679 leader_election.cc:302] T 21621e14f640407ab71cb3e55c76dc36 P 7676cbdb92e040f9b34aef99c0ddbc30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7676cbdb92e040f9b34aef99c0ddbc30, 7882063db22d46a39672797ac9638ce4; no voters:
I20260807 07:04:56.835352 10232 raft_consensus.cc:2781] T 21621e14f640407ab71cb3e55c76dc36 P 7676cbdb92e040f9b34aef99c0ddbc30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 07:04:56.835533 10232 raft_consensus.cc:482] T 21621e14f640407ab71cb3e55c76dc36 P 7676cbdb92e040f9b34aef99c0ddbc30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:04:56.835667 10232 raft_consensus.cc:3037] T 21621e14f640407ab71cb3e55c76dc36 P 7676cbdb92e040f9b34aef99c0ddbc30 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:04:56.838949 10232 raft_consensus.cc:504] T 21621e14f640407ab71cb3e55c76dc36 P 7676cbdb92e040f9b34aef99c0ddbc30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe73acb9913948e09ae60afc1ebdf24f" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 45093 } } peers { permanent_uuid: "7882063db22d46a39672797ac9638ce4" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 45141 } } peers { permanent_uuid: "7676cbdb92e040f9b34aef99c0ddbc30" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 45401 } }
I20260807 07:04:56.839727 10232 leader_election.cc:288] T 21621e14f640407ab71cb3e55c76dc36 P 7676cbdb92e040f9b34aef99c0ddbc30 [CANDIDATE]: Term 1 election: Requested vote from peers fe73acb9913948e09ae60afc1ebdf24f (127.21.83.2:45093), 7882063db22d46a39672797ac9638ce4 (127.21.83.3:45141)
I20260807 07:04:56.840250 9948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21621e14f640407ab71cb3e55c76dc36" candidate_uuid: "7676cbdb92e040f9b34aef99c0ddbc30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe73acb9913948e09ae60afc1ebdf24f"
I20260807 07:04:56.840420 10127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21621e14f640407ab71cb3e55c76dc36" candidate_uuid: "7676cbdb92e040f9b34aef99c0ddbc30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7882063db22d46a39672797ac9638ce4"
I20260807 07:04:56.840507 9948 raft_consensus.cc:3037] T 21621e14f640407ab71cb3e55c76dc36 P fe73acb9913948e09ae60afc1ebdf24f [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:04:56.840656 10127 raft_consensus.cc:3037] T 21621e14f640407ab71cb3e55c76dc36 P 7882063db22d46a39672797ac9638ce4 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:04:56.843969 9948 raft_consensus.cc:2454] T 21621e14f640407ab71cb3e55c76dc36 P fe73acb9913948e09ae60afc1ebdf24f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7676cbdb92e040f9b34aef99c0ddbc30 in term 1.
I20260807 07:04:56.843958 10127 raft_consensus.cc:2454] T 21621e14f640407ab71cb3e55c76dc36 P 7882063db22d46a39672797ac9638ce4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7676cbdb92e040f9b34aef99c0ddbc30 in term 1.
I20260807 07:04:56.844473 9679 leader_election.cc:302] T 21621e14f640407ab71cb3e55c76dc36 P 7676cbdb92e040f9b34aef99c0ddbc30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7676cbdb92e040f9b34aef99c0ddbc30, 7882063db22d46a39672797ac9638ce4; no voters:
I20260807 07:04:56.844841 10232 raft_consensus.cc:2781] T 21621e14f640407ab71cb3e55c76dc36 P 7676cbdb92e040f9b34aef99c0ddbc30 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:04:56.845429 10232 raft_consensus.cc:686] T 21621e14f640407ab71cb3e55c76dc36 P 7676cbdb92e040f9b34aef99c0ddbc30 [term 1 LEADER]: Becoming Leader. State: Replica: 7676cbdb92e040f9b34aef99c0ddbc30, State: Running, Role: LEADER
I20260807 07:04:56.846062 10232 consensus_queue.cc:237] T 21621e14f640407ab71cb3e55c76dc36 P 7676cbdb92e040f9b34aef99c0ddbc30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe73acb9913948e09ae60afc1ebdf24f" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 45093 } } peers { permanent_uuid: "7882063db22d46a39672797ac9638ce4" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 45141 } } peers { permanent_uuid: "7676cbdb92e040f9b34aef99c0ddbc30" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 45401 } }
I20260807 07:04:56.851215 9568 catalog_manager.cc:5495] T 21621e14f640407ab71cb3e55c76dc36 P 7676cbdb92e040f9b34aef99c0ddbc30 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7676cbdb92e040f9b34aef99c0ddbc30 (127.21.83.1). New cstate: current_term: 1 leader_uuid: "7676cbdb92e040f9b34aef99c0ddbc30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe73acb9913948e09ae60afc1ebdf24f" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 45093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7882063db22d46a39672797ac9638ce4" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 45141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7676cbdb92e040f9b34aef99c0ddbc30" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 45401 } health_report { overall_health: HEALTHY } } }
I20260807 07:04:56.917343 10218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:56.911584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52888 (local address 127.21.83.1:45401)
0807 07:04:56.911760 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:56.911765 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:56.911780 (+ 15us) server_negotiation.cc:407] Connection header received
0807 07:04:56.911870 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:56.911874 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:56.911951 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:56.912083 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:04:56.912848 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:56.913672 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:56.915229 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:56.915676 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:56.915775 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:56.916237 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:04:56.916764 (+ 527us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:04:56.916881 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:56.917062 (+ 181us) server_negotiation.cc:299] Negotiation successful
0807 07:04:56.917138 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260807 07:04:56.926034 10127 raft_consensus.cc:1261] T 21621e14f640407ab71cb3e55c76dc36 P 7882063db22d46a39672797ac9638ce4 [term 1 FOLLOWER]: Refusing update from remote peer 7676cbdb92e040f9b34aef99c0ddbc30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:04:56.926049 9948 raft_consensus.cc:1261] T 21621e14f640407ab71cb3e55c76dc36 P fe73acb9913948e09ae60afc1ebdf24f [term 1 FOLLOWER]: Refusing update from remote peer 7676cbdb92e040f9b34aef99c0ddbc30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:04:56.927214 10236 consensus_queue.cc:1035] T 21621e14f640407ab71cb3e55c76dc36 P 7676cbdb92e040f9b34aef99c0ddbc30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7882063db22d46a39672797ac9638ce4" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 45141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:04:56.927611 10232 consensus_queue.cc:1035] T 21621e14f640407ab71cb3e55c76dc36 P 7676cbdb92e040f9b34aef99c0ddbc30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe73acb9913948e09ae60afc1ebdf24f" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 45093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:04:56.956975 9568 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:04:56.957264 9568 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:04:56.961009 9568 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad: Sending DeleteTablet for 3 replicas of tablet 21621e14f640407ab71cb3e55c76dc36
I20260807 07:04:56.962152 9917 tablet_service.cc:1514] Processing DeleteTablet for tablet 21621e14f640407ab71cb3e55c76dc36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:04:56 UTC) from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:43306
I20260807 07:04:56.962324 10098 tablet_service.cc:1514] Processing DeleteTablet for tablet 21621e14f640407ab71cb3e55c76dc36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:04:56 UTC) from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:42432
I20260807 07:04:56.962568 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 9650
I20260807 07:04:56.962540 9740 tablet_service.cc:1514] Processing DeleteTablet for tablet 21621e14f640407ab71cb3e55c76dc36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:04:56 UTC) from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:52876
I20260807 07:04:56.973443 10256 tablet_replica.cc:330] T 21621e14f640407ab71cb3e55c76dc36 P 7676cbdb92e040f9b34aef99c0ddbc30: stopping tablet replica
I20260807 07:04:56.973608 10253 tablet_replica.cc:330] T 21621e14f640407ab71cb3e55c76dc36 P fe73acb9913948e09ae60afc1ebdf24f: stopping tablet replica
I20260807 07:04:56.973924 10256 raft_consensus.cc:2229] T 21621e14f640407ab71cb3e55c76dc36 P 7676cbdb92e040f9b34aef99c0ddbc30 [term 1 LEADER]: Raft consensus shutting down.
I20260807 07:04:56.974000 10253 raft_consensus.cc:2229] T 21621e14f640407ab71cb3e55c76dc36 P fe73acb9913948e09ae60afc1ebdf24f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:04:56.974640 10253 raft_consensus.cc:2258] T 21621e14f640407ab71cb3e55c76dc36 P fe73acb9913948e09ae60afc1ebdf24f [term 1 FOLLOWER]: Raft consensus is shut down!
Aug 07 07:04:56 dist-test-slave-ncp9 krb5kdc[9502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.1@KRBTEST.COM
I20260807 07:04:56.976848 10253 ts_tablet_manager.cc:1905] T 21621e14f640407ab71cb3e55c76dc36 P fe73acb9913948e09ae60afc1ebdf24f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:04:56.976357 10256 raft_consensus.cc:2258] T 21621e14f640407ab71cb3e55c76dc36 P 7676cbdb92e040f9b34aef99c0ddbc30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:04:56.980672 10256 ts_tablet_manager.cc:1905] T 21621e14f640407ab71cb3e55c76dc36 P 7676cbdb92e040f9b34aef99c0ddbc30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:04:56.980825 10254 tablet_replica.cc:330] T 21621e14f640407ab71cb3e55c76dc36 P 7882063db22d46a39672797ac9638ce4: stopping tablet replica
I20260807 07:04:56.981375 10254 raft_consensus.cc:2229] T 21621e14f640407ab71cb3e55c76dc36 P 7882063db22d46a39672797ac9638ce4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:04:56.981972 10254 raft_consensus.cc:2258] T 21621e14f640407ab71cb3e55c76dc36 P 7882063db22d46a39672797ac9638ce4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:04:56.983856 10218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:56.963573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52898 (local address 127.21.83.1:45401)
0807 07:04:56.963796 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:56.963802 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:56.963873 (+ 71us) server_negotiation.cc:407] Connection header received
0807 07:04:56.964168 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:56.964175 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:56.964270 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:56.964341 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:04:56.966387 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:56.967629 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:56.969734 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:56.970492 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:56.970596 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:56.976055 (+ 5459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:04:56.976069 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:04:56.976098 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:04:56.976149 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:04:56.980069 (+ 3920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:56.981113 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:56.981122 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:56.981125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:56.981298 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:56.981790 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:56.981795 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:56.981799 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:56.982308 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:04:56.982627 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:56.983238 (+ 611us) server_negotiation.cc:299] Negotiation successful
0807 07:04:56.983484 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 07:04:56.985483 9815 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:04:56.985872 10253 ts_tablet_manager.cc:1918] T 21621e14f640407ab71cb3e55c76dc36 P fe73acb9913948e09ae60afc1ebdf24f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:04:56.987674 10254 ts_tablet_manager.cc:1905] T 21621e14f640407ab71cb3e55c76dc36 P 7882063db22d46a39672797ac9638ce4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:04:56.986244 10253 log.cc:1196] T 21621e14f640407ab71cb3e55c76dc36 P fe73acb9913948e09ae60afc1ebdf24f: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-1/wal/wals/21621e14f640407ab71cb3e55c76dc36
I20260807 07:04:56.989871 10253 ts_tablet_manager.cc:1939] T 21621e14f640407ab71cb3e55c76dc36 P fe73acb9913948e09ae60afc1ebdf24f: Deleting consensus metadata
I20260807 07:04:56.991701 9544 catalog_manager.cc:4841] TS fe73acb9913948e09ae60afc1ebdf24f (127.21.83.2:45093): tablet 21621e14f640407ab71cb3e55c76dc36 (table test-table [id=3a7f26eedfbe49da952fa5ad0352fb0f]) successfully deleted
I20260807 07:04:56.993858 10254 ts_tablet_manager.cc:1918] T 21621e14f640407ab71cb3e55c76dc36 P 7882063db22d46a39672797ac9638ce4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:04:56.993989 10254 log.cc:1196] T 21621e14f640407ab71cb3e55c76dc36 P 7882063db22d46a39672797ac9638ce4: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-2/wal/wals/21621e14f640407ab71cb3e55c76dc36
I20260807 07:04:56.994419 10254 ts_tablet_manager.cc:1939] T 21621e14f640407ab71cb3e55c76dc36 P 7882063db22d46a39672797ac9638ce4: Deleting consensus metadata
I20260807 07:04:56.995807 9544 catalog_manager.cc:4841] TS 7882063db22d46a39672797ac9638ce4 (127.21.83.3:45141): tablet 21621e14f640407ab71cb3e55c76dc36 (table test-table [id=3a7f26eedfbe49da952fa5ad0352fb0f]) successfully deleted
I20260807 07:04:57.082319 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 9650
I20260807 07:04:57.106137 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 9832
W20260807 07:04:57.106112 9545 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.83.1:45401 (error 108)
I20260807 07:04:57.108268 10264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:57.107465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:45401 (local address 127.0.0.1:52904)
0807 07:04:57.107906 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:57.108034 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.1:45401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":222,"threads_started":1}
W20260807 07:04:57.108546 9545 catalog_manager.cc:4584] TS 7676cbdb92e040f9b34aef99c0ddbc30 (127.21.83.1:45401): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 21621e14f640407ab71cb3e55c76dc36: Network error: Client connection negotiation failed: client connection to 127.21.83.1:45401: connect: Connection refused (error 111)
Aug 07 07:04:57 dist-test-slave-ncp9 krb5kdc[9502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.2@KRBTEST.COM
I20260807 07:04:57.121397 10214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:57.107002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43318 (local address 127.21.83.2:45093)
0807 07:04:57.107195 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:57.107201 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:57.107260 (+ 59us) server_negotiation.cc:407] Connection header received
0807 07:04:57.107456 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:57.107462 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:57.107562 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:57.107685 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:04:57.109196 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:57.109948 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:57.111184 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:57.111587 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:57.111674 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:57.115984 (+ 4310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:04:57.116004 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:04:57.116020 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:04:57.116066 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:04:57.118707 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:57.119451 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:57.119458 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:57.119460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:57.119609 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:57.119911 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:57.119914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:57.119916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:57.120289 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:04:57.120522 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:57.120809 (+ 287us) server_negotiation.cc:299] Negotiation successful
0807 07:04:57.121054 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 07:04:57.122125 10001 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:04:57.162142 10264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:57.161671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:45401 (local address 127.0.0.1:52916)
0807 07:04:57.161889 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:57.161992 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.1:45401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260807 07:04:57.163249 10264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:57.162901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:45401 (local address 127.0.0.1:52920)
0807 07:04:57.163057 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:57.163127 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.1:45401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260807 07:04:57.211246 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 9832
I20260807 07:04:57.227522 10264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:57.223667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:45401 (local address 127.0.0.1:52922)
0807 07:04:57.227098 (+ 3431us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:57.227287 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.1:45401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3159}
I20260807 07:04:57.228716 10264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:57.228350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:45401 (local address 127.0.0.1:52924)
0807 07:04:57.228478 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:57.228551 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.1:45401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260807 07:04:57.237908 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 10012
Aug 07 07:04:57 dist-test-slave-ncp9 krb5kdc[9502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.3@KRBTEST.COM
I20260807 07:04:57.252152 10219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:57.238594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42438 (local address 127.21.83.3:45141)
0807 07:04:57.238781 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:57.238787 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:57.238806 (+ 19us) server_negotiation.cc:407] Connection header received
0807 07:04:57.238896 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:57.238900 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:57.238966 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:57.239061 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:04:57.240325 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:57.241065 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:57.242371 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:57.242765 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:57.242870 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:57.247108 (+ 4238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:04:57.247121 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:04:57.247135 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:04:57.247167 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:04:57.249593 (+ 2426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:57.250228 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:57.250236 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:57.250238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:57.250371 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:57.250712 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:57.250716 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:57.250717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:57.251049 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:04:57.251326 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:57.251614 (+ 288us) server_negotiation.cc:299] Negotiation successful
0807 07:04:57.251831 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 07:04:57.252779 10188 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:04:57.309329 10264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:57.308856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:45401 (local address 127.0.0.1:52940)
0807 07:04:57.309089 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:57.309177 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.1:45401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260807 07:04:57.310331 10264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:57.309937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:45401 (local address 127.0.0.1:52948)
0807 07:04:57.310138 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:57.310206 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.1:45401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260807 07:04:57.342129 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 10012
I20260807 07:04:57.363479 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 9519
I20260807 07:04:57.364631 9607 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:04:57.479990 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 9519
I20260807 07:04:57.509872 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:40691
--webserver_interface=127.21.83.62
--webserver_port=39817
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:37735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:40691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:04:57.861328 10269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:04:57.861653 10269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:04:57.861793 10269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:04:57.869225 10269 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:04:57.869350 10269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:04:57.869431 10269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:04:57.869520 10269 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:04:57.869581 10269 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:04:57.879660 10269 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:37735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:40691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:40691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.83.62
--webserver_port=39817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:04:57.881012 10269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:04:57.882331 10269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:04:57.908156 10277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:04:57.908247 10276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:04:57.908195 10280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:04:57.909402 10269 server_base.cc:1034] running on GCE node
I20260807 07:04:57.910380 10269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:04:57.912130 10269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:04:57.913348 10269 hybrid_clock.cc:648] HybridClock initialized: now 1786086297913282 us; error 65 us; skew 500 ppm
I20260807 07:04:57.913774 10269 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:04:57 dist-test-slave-ncp9 krb5kdc[9502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086297, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:04:57.921175 10269 init.cc:377] Logged in from keytab as kudu/127.21.83.62@KRBTEST.COM (short username kudu)
I20260807 07:04:57.922971 10269 webserver.cc:460] Webserver started at http://127.21.83.62:39817/ using document root <none> and password file <none>
I20260807 07:04:57.923733 10269 fs_manager.cc:359] Metadata directory not provided
I20260807 07:04:57.923875 10269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:04:57.929864 10269 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260807 07:04:57.933048 10291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:04:57.934569 10269 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 07:04:57.934770 10269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "189302c9464749e7a638be128648baad"
format_stamp: "Formatted at 2026-08-07 07:04:54 on dist-test-slave-ncp9"
server_key: "8361948d5d4132a9cd9b764014e18406"
server_key_iv: "e35ee20c7aec6fa64cede26b417bf78d"
server_key_version: "encryptionkey@0"
I20260807 07:04:57.936460 10269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:04:57.966634 10269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:04:57.971468 10269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:04:57.971777 10269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:04:57.991770 10269 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.62:40691
I20260807 07:04:57.991791 10360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.62:40691 every 8 connection(s)
I20260807 07:04:57.993469 10269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:04:58.001241 10361 sys_catalog.cc:263] Verifying existing consensus state
I20260807 07:04:58.003046 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 10269
I20260807 07:04:58.003938 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.1:45401
--local_ip_for_outbound_sockets=127.21.83.1
--tserver_master_addrs=127.21.83.62:40691
--webserver_port=38133
--webserver_interface=127.21.83.1
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab
--principal=kudu/127.21.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:37735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 07:04:58.005331 10361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad: Bootstrap starting.
I20260807 07:04:58.027848 10361 log.cc:824] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad: Log is configured to *not* fsync() on all Append() calls
I20260807 07:04:58.041167 10361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 07:04:58.041811 10361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad: Bootstrap complete.
I20260807 07:04:58.051746 10361 raft_consensus.cc:348] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "189302c9464749e7a638be128648baad" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 40691 } }
I20260807 07:04:58.052682 10361 raft_consensus.cc:729] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 189302c9464749e7a638be128648baad, State: Initialized, Role: FOLLOWER
I20260807 07:04:58.053424 10361 consensus_queue.cc:260] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "189302c9464749e7a638be128648baad" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 40691 } }
I20260807 07:04:58.053687 10361 raft_consensus.cc:388] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:04:58.053810 10361 raft_consensus.cc:482] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:04:58.053969 10361 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad [term 1 FOLLOWER]: Advancing to term 2
I20260807 07:04:58.058145 10361 raft_consensus.cc:504] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "189302c9464749e7a638be128648baad" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 40691 } }
I20260807 07:04:58.058727 10361 leader_election.cc:302] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 189302c9464749e7a638be128648baad; no voters:
I20260807 07:04:58.059355 10361 leader_election.cc:288] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad [CANDIDATE]: Term 2 election: Requested vote from peers
I20260807 07:04:58.059499 10365 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad [term 2 FOLLOWER]: Leader election won for term 2
I20260807 07:04:58.060765 10365 raft_consensus.cc:686] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad [term 2 LEADER]: Becoming Leader. State: Replica: 189302c9464749e7a638be128648baad, State: Running, Role: LEADER
I20260807 07:04:58.061488 10365 consensus_queue.cc:237] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "189302c9464749e7a638be128648baad" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 40691 } }
I20260807 07:04:58.061841 10361 sys_catalog.cc:564] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:04:58.064524 10368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 189302c9464749e7a638be128648baad. Latest consensus state: current_term: 2 leader_uuid: "189302c9464749e7a638be128648baad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "189302c9464749e7a638be128648baad" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 40691 } } }
I20260807 07:04:58.065310 10368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad [sys.catalog]: This master's current role is: LEADER
I20260807 07:04:58.064517 10367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "189302c9464749e7a638be128648baad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "189302c9464749e7a638be128648baad" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 40691 } } }
I20260807 07:04:58.065968 10367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad [sys.catalog]: This master's current role is: LEADER
I20260807 07:04:58.078552 10376 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:04:58.086824 10376 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:04:58.088420 10376 catalog_manager.cc:1259] Loaded cluster ID: f014a38f54fe49d19698d571a1c22ed8
I20260807 07:04:58.088574 10376 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:04:58.091969 10376 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260807 07:04:58.111096 10376 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260807 07:04:58.121021 10376 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:04:58.122833 10376 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad: Loaded TSK: 0
I20260807 07:04:58.124344 10376 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 07:04:58.375672 10363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:04:58.375958 10363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:04:58.376078 10363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:04:58.383460 10363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:04:58.383576 10363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:04:58.383723 10363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.1
I20260807 07:04:58.393661 10363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:37735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab
--principal=kudu/127.21.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.1:45401
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.83.1
--webserver_port=38133
--tserver_master_addrs=127.21.83.62:40691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.1
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:04:58.394991 10363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:04:58.396313 10363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:04:58.409880 10405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:04:58.410250 10406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:04:58.410925 10411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:04:58.411022 10363 server_base.cc:1034] running on GCE node
I20260807 07:04:58.411806 10363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:04:58.412988 10363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:04:58.414224 10363 hybrid_clock.cc:648] HybridClock initialized: now 1786086298414172 us; error 38 us; skew 500 ppm
I20260807 07:04:58.415205 10363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:04:58 dist-test-slave-ncp9 krb5kdc[9502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086298, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:04:58.419384 10363 init.cc:377] Logged in from keytab as kudu/127.21.83.1@KRBTEST.COM (short username kudu)
I20260807 07:04:58.421195 10363 webserver.cc:460] Webserver started at http://127.21.83.1:38133/ using document root <none> and password file <none>
I20260807 07:04:58.422073 10363 fs_manager.cc:359] Metadata directory not provided
I20260807 07:04:58.422259 10363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:04:58.429414 10363 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.002s
I20260807 07:04:58.433035 10420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:04:58.434532 10363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 07:04:58.434711 10363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-0/wal
uuid: "7676cbdb92e040f9b34aef99c0ddbc30"
format_stamp: "Formatted at 2026-08-07 07:04:55 on dist-test-slave-ncp9"
server_key: "c2205ded58806d934c6bd252549406ab"
server_key_iv: "d5f9ea3655889aff94521f9360313c0d"
server_key_version: "encryptionkey@0"
I20260807 07:04:58.435750 10363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:04:58.476045 10363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:04:58.480286 10363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:04:58.480558 10363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:04:58.481638 10363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:04:58.483556 10428 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260807 07:04:58.488962 10428 ts_tablet_manager.cc:1852] T 21621e14f640407ab71cb3e55c76dc36 P 7676cbdb92e040f9b34aef99c0ddbc30: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260807 07:04:58.489972 10428 ts_tablet_manager.cc:1905] T 21621e14f640407ab71cb3e55c76dc36 P 7676cbdb92e040f9b34aef99c0ddbc30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:04:58.497296 10428 ts_tablet_manager.cc:1918] T 21621e14f640407ab71cb3e55c76dc36 P 7676cbdb92e040f9b34aef99c0ddbc30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:04:58.497494 10428 log.cc:1196] T 21621e14f640407ab71cb3e55c76dc36 P 7676cbdb92e040f9b34aef99c0ddbc30: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-0/wal/wals/21621e14f640407ab71cb3e55c76dc36
I20260807 07:04:58.497952 10428 ts_tablet_manager.cc:1939] T 21621e14f640407ab71cb3e55c76dc36 P 7676cbdb92e040f9b34aef99c0ddbc30: Deleting consensus metadata
I20260807 07:04:58.498324 10363 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260807 07:04:58.498441 10363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.001s sys 0.000s
I20260807 07:04:58.498558 10363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:04:58.498628 10363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:04:58.530006 10363 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.1:45401
I20260807 07:04:58.530071 10566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.1:45401 every 8 connection(s)
I20260807 07:04:58.531273 10363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
I20260807 07:04:58.537518 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 10363
I20260807 07:04:58.538300 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.2:45093
--local_ip_for_outbound_sockets=127.21.83.2
--tserver_master_addrs=127.21.83.62:40691
--webserver_port=36915
--webserver_interface=127.21.83.2
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab
--principal=kudu/127.21.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:37735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 07:04:58 dist-test-slave-ncp9 krb5kdc[9502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086298, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.1@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:04:58.557552 10573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:58.534006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:57079 (local address 127.21.83.62:40691)
0807 07:04:58.534594 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:58.534605 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:58.535293 (+ 688us) server_negotiation.cc:407] Connection header received
0807 07:04:58.537712 (+ 2419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:58.537731 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:58.538229 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:58.538692 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:04:58.540341 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:58.542224 (+ 1883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:58.543964 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:58.544983 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:58.545202 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:58.549075 (+ 3873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:04:58.549087 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:04:58.549109 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:04:58.549171 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:04:58.552632 (+ 3461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:58.553728 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:58.553736 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:58.553741 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:58.553942 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:58.554493 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:58.554497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:58.554500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:58.555059 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:04:58.555217 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:58.556176 (+ 959us) server_negotiation.cc:299] Negotiation successful
0807 07:04:58.556516 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":265,"threads_started":1}
I20260807 07:04:58.557781 10574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:58.534261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.1:57079)
0807 07:04:58.535102 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:58.535152 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0807 07:04:58.537042 (+ 1890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:04:58.538802 (+ 1760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:04:58.538810 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:04:58.539418 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:04:58.540038 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:58.540060 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:58.542442 (+ 2382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:58.542446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:58.543771 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:58.543787 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:58.545244 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:58.545250 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:58.545462 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:58.546189 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:04:58.546220 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:04:58.548693 (+ 2473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:04:58.552844 (+ 4151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:04:58.552854 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:04:58.552872 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:04:58.553550 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:04:58.554083 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:04:58.554088 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:04:58.554092 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:04:58.554345 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:04:58.555244 (+ 899us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:04:58.555254 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:04:58.555685 (+ 431us) client_negotiation.cc:769] Sending connection context
0807 07:04:58.555982 (+ 297us) client_negotiation.cc:240] Negotiation successful
0807 07:04:58.556520 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":342,"threads_started":1}
I20260807 07:04:58.560910 10567 heartbeater.cc:347] Connected to a master server at 127.21.83.62:40691
I20260807 07:04:58.561225 10567 heartbeater.cc:464] Registering TS with master...
I20260807 07:04:58.561982 10567 heartbeater.cc:511] Master 127.21.83.62:40691 requested a full tablet report, sending...
I20260807 07:04:58.564252 10314 ts_manager.cc:194] Registered new tserver with Master: 7676cbdb92e040f9b34aef99c0ddbc30 (127.21.83.1:45401)
I20260807 07:04:58.566262 10314 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.1@KRBTEST.COM'} at 127.21.83.1:57079
W20260807 07:04:58.877321 10575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:04:58.877612 10575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:04:58.877719 10575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:04:58.885067 10575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:04:58.885191 10575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:04:58.885308 10575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.2
I20260807 07:04:58.895099 10575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:37735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab
--principal=kudu/127.21.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.2:45093
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.83.2
--webserver_port=36915
--tserver_master_addrs=127.21.83.62:40691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.2
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:04:58.896354 10575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:04:58.897656 10575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:04:58.910264 10582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:04:58.910840 10584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:04:58.910841 10586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:04:58.910908 10575 server_base.cc:1034] running on GCE node
I20260807 07:04:58.911574 10575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:04:58.912425 10575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:04:58.913625 10575 hybrid_clock.cc:648] HybridClock initialized: now 1786086298913561 us; error 43 us; skew 500 ppm
I20260807 07:04:58.913909 10575 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:04:58 dist-test-slave-ncp9 krb5kdc[9502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086298, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:04:58.917064 10575 init.cc:377] Logged in from keytab as kudu/127.21.83.2@KRBTEST.COM (short username kudu)
I20260807 07:04:58.918349 10575 webserver.cc:460] Webserver started at http://127.21.83.2:36915/ using document root <none> and password file <none>
I20260807 07:04:58.918972 10575 fs_manager.cc:359] Metadata directory not provided
I20260807 07:04:58.919104 10575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:04:58.924113 10575 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260807 07:04:58.926683 10595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:04:58.927871 10575 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260807 07:04:58.928011 10575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-1/wal
uuid: "fe73acb9913948e09ae60afc1ebdf24f"
format_stamp: "Formatted at 2026-08-07 07:04:55 on dist-test-slave-ncp9"
server_key: "10d60cbdc41369e1502319772e11b105"
server_key_iv: "a99087ea8866ef68b76a9cf1039ce550"
server_key_version: "encryptionkey@0"
I20260807 07:04:58.928957 10575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:04:58.944290 10575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:04:58.948050 10575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:04:58.948293 10575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:04:58.949285 10575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:04:58.950533 10575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:04:58.950639 10575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:04:58.950743 10575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:04:58.950829 10575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:04:58.980726 10575 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.2:45093
I20260807 07:04:58.980785 10768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.2:45093 every 8 connection(s)
I20260807 07:04:58.982097 10575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
I20260807 07:04:58.983028 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 10575
I20260807 07:04:58.983799 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.3:45141
--local_ip_for_outbound_sockets=127.21.83.3
--tserver_master_addrs=127.21.83.62:40691
--webserver_port=33465
--webserver_interface=127.21.83.3
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab
--principal=kudu/127.21.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:37735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 07:04:58 dist-test-slave-ncp9 krb5kdc[9502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086298, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.2@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:04:59.003906 10573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:58.984797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:51719 (local address 127.21.83.62:40691)
0807 07:04:58.985025 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:58.985031 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:58.986240 (+ 1209us) server_negotiation.cc:407] Connection header received
0807 07:04:58.988440 (+ 2200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:58.988444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:58.988512 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:58.988595 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:04:58.990445 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:58.991371 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:58.993372 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:58.994061 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:58.994213 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:58.997863 (+ 3650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:04:58.997871 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:04:58.997876 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:04:58.997916 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:04:59.000630 (+ 2714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:59.001372 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:59.001376 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:59.001378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:59.001515 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:59.002066 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:59.002070 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:59.002072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:59.002395 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:04:59.002548 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:59.003297 (+ 749us) server_negotiation.cc:299] Negotiation successful
0807 07:04:59.003546 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 07:04:59.005019 10772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:58.985276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.2:51719)
0807 07:04:58.986065 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:58.986108 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 07:04:58.988184 (+ 2076us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:04:58.988844 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:04:58.988855 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:04:58.989514 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:04:58.990267 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:58.990290 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:58.991524 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:58.991529 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:58.993218 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:58.993234 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:58.994296 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:58.994302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:58.994568 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:58.995323 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:04:58.995350 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:04:58.997646 (+ 2296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:04:59.000768 (+ 3122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:04:59.000776 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:04:59.000791 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:04:59.001255 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:04:59.001617 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:04:59.001622 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:04:59.001626 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:04:59.001955 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:04:59.002520 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:04:59.002527 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:04:59.002868 (+ 341us) client_negotiation.cc:769] Sending connection context
0807 07:04:59.003165 (+ 297us) client_negotiation.cc:240] Negotiation successful
0807 07:04:59.003750 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":309,"threads_started":1}
I20260807 07:04:59.006459 10770 heartbeater.cc:347] Connected to a master server at 127.21.83.62:40691
I20260807 07:04:59.006745 10770 heartbeater.cc:464] Registering TS with master...
I20260807 07:04:59.007439 10770 heartbeater.cc:511] Master 127.21.83.62:40691 requested a full tablet report, sending...
I20260807 07:04:59.008777 10314 ts_manager.cc:194] Registered new tserver with Master: fe73acb9913948e09ae60afc1ebdf24f (127.21.83.2:45093)
I20260807 07:04:59.009637 10314 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.2@KRBTEST.COM'} at 127.21.83.2:51719
W20260807 07:04:59.317909 10773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:04:59.318230 10773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:04:59.318333 10773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:04:59.326167 10773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:04:59.326287 10773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:04:59.326434 10773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.3
I20260807 07:04:59.336148 10773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:37735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab
--principal=kudu/127.21.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.3:45141
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.83.3
--webserver_port=33465
--tserver_master_addrs=127.21.83.62:40691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.3
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:04:59.337330 10773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:04:59.338593 10773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:04:59.350675 10784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:04:59.350762 10782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:04:59.350941 10786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:04:59.351673 10773 server_base.cc:1034] running on GCE node
I20260807 07:04:59.352202 10773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:04:59.353055 10773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:04:59.354246 10773 hybrid_clock.cc:648] HybridClock initialized: now 1786086299354206 us; error 39 us; skew 500 ppm
I20260807 07:04:59.354519 10773 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:04:59 dist-test-slave-ncp9 krb5kdc[9502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086299, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:04:59.357432 10773 init.cc:377] Logged in from keytab as kudu/127.21.83.3@KRBTEST.COM (short username kudu)
I20260807 07:04:59.358788 10773 webserver.cc:460] Webserver started at http://127.21.83.3:33465/ using document root <none> and password file <none>
I20260807 07:04:59.359390 10773 fs_manager.cc:359] Metadata directory not provided
I20260807 07:04:59.359515 10773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:04:59.364562 10773 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260807 07:04:59.367198 10798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:04:59.368608 10773 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 07:04:59.368767 10773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-2/wal
uuid: "7882063db22d46a39672797ac9638ce4"
format_stamp: "Formatted at 2026-08-07 07:04:56 on dist-test-slave-ncp9"
server_key: "9a7eedc45af41856860fbb6aef7e54ba"
server_key_iv: "bb8699a76a13c87918cb06bbfc46f5f9"
server_key_version: "encryptionkey@0"
I20260807 07:04:59.369820 10773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:04:59.386914 10773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:04:59.390473 10773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:04:59.390717 10773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:04:59.391657 10773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:04:59.392927 10773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:04:59.393025 10773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:04:59.393131 10773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:04:59.393200 10773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:04:59.424715 10773 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.3:45141
I20260807 07:04:59.424779 10954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.3:45141 every 8 connection(s)
I20260807 07:04:59.426126 10773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
I20260807 07:04:59.429164 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 10773
Aug 07 07:04:59 dist-test-slave-ncp9 krb5kdc[9502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086299, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.3@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:04:59.445784 10959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:59.430159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50788 (local address 127.21.83.62:40691)
0807 07:04:59.430612 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:59.430618 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:59.431100 (+ 482us) server_negotiation.cc:407] Connection header received
0807 07:04:59.431273 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:59.431278 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:59.431353 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:59.431441 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:04:59.433367 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:59.434271 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:59.435952 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:59.436385 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:59.436475 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:59.439153 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:04:59.439158 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:04:59.439161 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:04:59.439192 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:04:59.442222 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:59.443010 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:59.443016 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:59.443037 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:59.443220 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:59.443673 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:59.443678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:59.443682 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:59.444065 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:04:59.444324 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:59.444813 (+ 489us) server_negotiation.cc:299] Negotiation successful
0807 07:04:59.445381 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":238,"threads_started":1}
I20260807 07:04:59.447223 10573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:59.428475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:55253 (local address 127.21.83.62:40691)
0807 07:04:59.428668 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:59.428672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:59.429845 (+ 1173us) server_negotiation.cc:407] Connection header received
0807 07:04:59.432167 (+ 2322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:59.432170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:59.432250 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:59.432320 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:04:59.433972 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:59.434778 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:59.436625 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:59.437079 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:59.437182 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:59.440538 (+ 3356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:04:59.440542 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:04:59.440545 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:04:59.440586 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:04:59.443683 (+ 3097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:59.444532 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:59.444537 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:59.444540 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:59.444700 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:04:59.445243 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:04:59.445248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:04:59.445252 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:04:59.445585 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:04:59.445716 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:59.446589 (+ 873us) server_negotiation.cc:299] Negotiation successful
0807 07:04:59.446783 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 07:04:59.448113 10958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:59.428865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.3:55253)
0807 07:04:59.429698 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:59.429742 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 07:04:59.431912 (+ 2170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:04:59.432581 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:04:59.432593 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:04:59.433274 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:04:59.433812 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:59.433827 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:59.434934 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:59.434940 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:59.436484 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:59.436498 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:59.437305 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:59.437311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:59.437530 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:59.438091 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:04:59.438112 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:04:59.440325 (+ 2213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:04:59.443830 (+ 3505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:04:59.443838 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:04:59.443851 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:04:59.444395 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:04:59.444843 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:04:59.444848 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:04:59.444851 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:04:59.445107 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:04:59.445786 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:04:59.445792 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:04:59.446166 (+ 374us) client_negotiation.cc:769] Sending connection context
0807 07:04:59.446436 (+ 270us) client_negotiation.cc:240] Negotiation successful
0807 07:04:59.446884 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":314,"threads_started":1}
I20260807 07:04:59.449945 10956 heartbeater.cc:347] Connected to a master server at 127.21.83.62:40691
I20260807 07:04:59.450202 10956 heartbeater.cc:464] Registering TS with master...
I20260807 07:04:59.450914 10956 heartbeater.cc:511] Master 127.21.83.62:40691 requested a full tablet report, sending...
I20260807 07:04:59.452337 10314 ts_manager.cc:194] Registered new tserver with Master: 7882063db22d46a39672797ac9638ce4 (127.21.83.3:45141)
I20260807 07:04:59.453164 10314 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.3@KRBTEST.COM'} at 127.21.83.3:55253
I20260807 07:04:59.461949 21836 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 07:04:59.468312 10573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:59.462844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50790 (local address 127.21.83.62:40691)
0807 07:04:59.462989 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:59.462993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:59.463506 (+ 513us) server_negotiation.cc:407] Connection header received
0807 07:04:59.463709 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:59.463712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:59.463844 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:59.463956 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:04:59.464312 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:59.465080 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:59.466513 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:59.466896 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:59.466978 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:59.467373 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:04:59.467882 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:04:59.467988 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:59.468061 (+ 73us) server_negotiation.cc:299] Negotiation successful
0807 07:04:59.468123 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260807 07:04:59.469161 10314 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:50790:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 07:04:59.476366 10314 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 07:04:59.499634 10972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:59.487934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:45401 (local address 127.0.0.1:34400)
0807 07:04:59.490103 (+ 2169us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:59.490117 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 07:04:59.490258 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:04:59.491185 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:04:59.491194 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:04:59.491225 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:04:59.491703 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:59.491716 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:59.493803 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:59.493809 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:59.496401 (+ 2592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:59.496417 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:59.498586 (+ 2169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:59.498592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:59.498786 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:59.498835 (+ 49us) client_negotiation.cc:769] Sending connection context
0807 07:04:59.498964 (+ 129us) client_negotiation.cc:240] Negotiation successful
0807 07:04:59.499164 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2066,"mutex_wait_us":3,"thread_start_us":1172,"threads_started":1}
I20260807 07:04:59.500458 10974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:59.487949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59972 (local address 127.21.83.2:45093)
0807 07:04:59.488639 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:59.488649 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:59.489705 (+ 1056us) server_negotiation.cc:407] Connection header received
0807 07:04:59.490585 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:59.490595 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:59.490840 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:59.491026 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:04:59.492153 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:59.493896 (+ 1743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:59.496732 (+ 2836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:59.498410 (+ 1678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:59.498543 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:59.498793 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:59.499452 (+ 659us) server_negotiation.cc:299] Negotiation successful
0807 07:04:59.499624 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":736,"threads_started":1}
I20260807 07:04:59.500582 10977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:59.488484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34400 (local address 127.21.83.1:45401)
0807 07:04:59.490138 (+ 1654us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:59.490147 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:59.490176 (+ 29us) server_negotiation.cc:407] Connection header received
0807 07:04:59.490525 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:59.490534 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:59.490762 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:59.490948 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:04:59.491960 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:59.493622 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:59.496605 (+ 2983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:59.498096 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:59.498231 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:59.498431 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:59.500038 (+ 1607us) server_negotiation.cc:299] Negotiation successful
0807 07:04:59.500222 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1498,"thread_start_us":1369,"threads_started":1}
I20260807 07:04:59.501721 10973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:59.487693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.2:45093 (local address 127.0.0.1:59972)
0807 07:04:59.489536 (+ 1843us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:59.489584 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 07:04:59.490286 (+ 702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:04:59.491389 (+ 1103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:04:59.491394 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:04:59.491427 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:04:59.491923 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:59.491938 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:59.494138 (+ 2200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:59.494144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:59.496509 (+ 2365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:59.496528 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:59.499017 (+ 2489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:59.499025 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:59.499202 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:59.499237 (+ 35us) client_negotiation.cc:769] Sending connection context
0807 07:04:59.501033 (+ 1796us) client_negotiation.cc:240] Negotiation successful
0807 07:04:59.501194 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1567,"mutex_wait_us":3,"thread_start_us":2176,"threads_started":1}
I20260807 07:04:59.502600 10480 tablet_service.cc:1467] Processing CreateTablet for tablet 7645630bc1984deeafb4c5983e6ebfdc (DEFAULT_TABLE table=test-table [id=e3b859aa8ce34e1c98632ddcd134a81e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:04:59.504005 10480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7645630bc1984deeafb4c5983e6ebfdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:04:59.506058 10667 tablet_service.cc:1467] Processing CreateTablet for tablet 7645630bc1984deeafb4c5983e6ebfdc (DEFAULT_TABLE table=test-table [id=e3b859aa8ce34e1c98632ddcd134a81e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:04:59.507275 10667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7645630bc1984deeafb4c5983e6ebfdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:04:59.509262 10972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:59.490279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.3:45141 (local address 127.0.0.1:41962)
0807 07:04:59.501693 (+ 11414us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:59.501715 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 07:04:59.501806 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:04:59.503050 (+ 1244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:04:59.503055 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:04:59.503084 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:04:59.503268 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:59.503281 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:59.505839 (+ 2558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:59.505844 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:59.507177 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:59.507191 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:59.508463 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:59.508470 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:59.508659 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:59.508695 (+ 36us) client_negotiation.cc:769] Sending connection context
0807 07:04:59.508788 (+ 93us) client_negotiation.cc:240] Negotiation successful
0807 07:04:59.508964 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11324,"thread_start_us":11975,"threads_started":1}
I20260807 07:04:59.509356 10980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:59.490548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41962 (local address 127.21.83.3:45141)
0807 07:04:59.502487 (+ 11939us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:59.502494 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:59.502526 (+ 32us) server_negotiation.cc:407] Connection header received
0807 07:04:59.502619 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:59.502627 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:59.502845 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:59.503024 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:04:59.504206 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:59.505625 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:59.507353 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:59.508286 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:59.508419 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:59.508663 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:59.508847 (+ 184us) server_negotiation.cc:299] Negotiation successful
0807 07:04:59.509033 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11770,"thread_start_us":11903,"threads_started":1}
I20260807 07:04:59.512933 10862 tablet_service.cc:1467] Processing CreateTablet for tablet 7645630bc1984deeafb4c5983e6ebfdc (DEFAULT_TABLE table=test-table [id=e3b859aa8ce34e1c98632ddcd134a81e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:04:59.514434 10981 tablet_bootstrap.cc:492] T 7645630bc1984deeafb4c5983e6ebfdc P 7676cbdb92e040f9b34aef99c0ddbc30: Bootstrap starting.
I20260807 07:04:59.514534 10862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7645630bc1984deeafb4c5983e6ebfdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:04:59.518699 10981 tablet_bootstrap.cc:654] T 7645630bc1984deeafb4c5983e6ebfdc P 7676cbdb92e040f9b34aef99c0ddbc30: Neither blocks nor log segments found. Creating new log.
I20260807 07:04:59.520062 10981 log.cc:824] T 7645630bc1984deeafb4c5983e6ebfdc P 7676cbdb92e040f9b34aef99c0ddbc30: Log is configured to *not* fsync() on all Append() calls
I20260807 07:04:59.522570 10983 tablet_bootstrap.cc:492] T 7645630bc1984deeafb4c5983e6ebfdc P fe73acb9913948e09ae60afc1ebdf24f: Bootstrap starting.
I20260807 07:04:59.524077 10981 tablet_bootstrap.cc:492] T 7645630bc1984deeafb4c5983e6ebfdc P 7676cbdb92e040f9b34aef99c0ddbc30: No bootstrap required, opened a new log
I20260807 07:04:59.524312 10981 ts_tablet_manager.cc:1397] T 7645630bc1984deeafb4c5983e6ebfdc P 7676cbdb92e040f9b34aef99c0ddbc30: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260807 07:04:59.526259 10983 tablet_bootstrap.cc:654] T 7645630bc1984deeafb4c5983e6ebfdc P fe73acb9913948e09ae60afc1ebdf24f: Neither blocks nor log segments found. Creating new log.
I20260807 07:04:59.527508 10983 log.cc:824] T 7645630bc1984deeafb4c5983e6ebfdc P fe73acb9913948e09ae60afc1ebdf24f: Log is configured to *not* fsync() on all Append() calls
I20260807 07:04:59.530645 10983 tablet_bootstrap.cc:492] T 7645630bc1984deeafb4c5983e6ebfdc P fe73acb9913948e09ae60afc1ebdf24f: No bootstrap required, opened a new log
I20260807 07:04:59.530933 10983 ts_tablet_manager.cc:1397] T 7645630bc1984deeafb4c5983e6ebfdc P fe73acb9913948e09ae60afc1ebdf24f: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260807 07:04:59.531774 10987 tablet_bootstrap.cc:492] T 7645630bc1984deeafb4c5983e6ebfdc P 7882063db22d46a39672797ac9638ce4: Bootstrap starting.
I20260807 07:04:59.533488 10981 raft_consensus.cc:348] T 7645630bc1984deeafb4c5983e6ebfdc P 7676cbdb92e040f9b34aef99c0ddbc30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe73acb9913948e09ae60afc1ebdf24f" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 45093 } } peers { permanent_uuid: "7676cbdb92e040f9b34aef99c0ddbc30" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 45401 } } peers { permanent_uuid: "7882063db22d46a39672797ac9638ce4" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 45141 } }
I20260807 07:04:59.533939 10981 raft_consensus.cc:374] T 7645630bc1984deeafb4c5983e6ebfdc P 7676cbdb92e040f9b34aef99c0ddbc30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:04:59.534046 10981 raft_consensus.cc:729] T 7645630bc1984deeafb4c5983e6ebfdc P 7676cbdb92e040f9b34aef99c0ddbc30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7676cbdb92e040f9b34aef99c0ddbc30, State: Initialized, Role: FOLLOWER
I20260807 07:04:59.534751 10981 consensus_queue.cc:260] T 7645630bc1984deeafb4c5983e6ebfdc P 7676cbdb92e040f9b34aef99c0ddbc30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe73acb9913948e09ae60afc1ebdf24f" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 45093 } } peers { permanent_uuid: "7676cbdb92e040f9b34aef99c0ddbc30" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 45401 } } peers { permanent_uuid: "7882063db22d46a39672797ac9638ce4" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 45141 } }
I20260807 07:04:59.535686 10987 tablet_bootstrap.cc:654] T 7645630bc1984deeafb4c5983e6ebfdc P 7882063db22d46a39672797ac9638ce4: Neither blocks nor log segments found. Creating new log.
I20260807 07:04:59.538215 10987 log.cc:824] T 7645630bc1984deeafb4c5983e6ebfdc P 7882063db22d46a39672797ac9638ce4: Log is configured to *not* fsync() on all Append() calls
I20260807 07:04:59.541821 10983 raft_consensus.cc:348] T 7645630bc1984deeafb4c5983e6ebfdc P fe73acb9913948e09ae60afc1ebdf24f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe73acb9913948e09ae60afc1ebdf24f" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 45093 } } peers { permanent_uuid: "7676cbdb92e040f9b34aef99c0ddbc30" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 45401 } } peers { permanent_uuid: "7882063db22d46a39672797ac9638ce4" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 45141 } }
I20260807 07:04:59.542272 10983 raft_consensus.cc:374] T 7645630bc1984deeafb4c5983e6ebfdc P fe73acb9913948e09ae60afc1ebdf24f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:04:59.542392 10983 raft_consensus.cc:729] T 7645630bc1984deeafb4c5983e6ebfdc P fe73acb9913948e09ae60afc1ebdf24f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe73acb9913948e09ae60afc1ebdf24f, State: Initialized, Role: FOLLOWER
I20260807 07:04:59.543017 10983 consensus_queue.cc:260] T 7645630bc1984deeafb4c5983e6ebfdc P fe73acb9913948e09ae60afc1ebdf24f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe73acb9913948e09ae60afc1ebdf24f" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 45093 } } peers { permanent_uuid: "7676cbdb92e040f9b34aef99c0ddbc30" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 45401 } } peers { permanent_uuid: "7882063db22d46a39672797ac9638ce4" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 45141 } }
I20260807 07:04:59.544255 10981 ts_tablet_manager.cc:1428] T 7645630bc1984deeafb4c5983e6ebfdc P 7676cbdb92e040f9b34aef99c0ddbc30: Time spent starting tablet: real 0.020s user 0.016s sys 0.004s
I20260807 07:04:59.544752 10567 heartbeater.cc:503] Master 127.21.83.62:40691 was elected leader, sending a full tablet report...
I20260807 07:04:59.545398 10770 heartbeater.cc:503] Master 127.21.83.62:40691 was elected leader, sending a full tablet report...
I20260807 07:04:59.545881 10987 tablet_bootstrap.cc:492] T 7645630bc1984deeafb4c5983e6ebfdc P 7882063db22d46a39672797ac9638ce4: No bootstrap required, opened a new log
I20260807 07:04:59.546036 10983 ts_tablet_manager.cc:1428] T 7645630bc1984deeafb4c5983e6ebfdc P fe73acb9913948e09ae60afc1ebdf24f: Time spent starting tablet: real 0.015s user 0.008s sys 0.007s
I20260807 07:04:59.546239 10987 ts_tablet_manager.cc:1397] T 7645630bc1984deeafb4c5983e6ebfdc P 7882063db22d46a39672797ac9638ce4: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.003s
I20260807 07:04:59.554510 10987 raft_consensus.cc:348] T 7645630bc1984deeafb4c5983e6ebfdc P 7882063db22d46a39672797ac9638ce4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe73acb9913948e09ae60afc1ebdf24f" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 45093 } } peers { permanent_uuid: "7676cbdb92e040f9b34aef99c0ddbc30" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 45401 } } peers { permanent_uuid: "7882063db22d46a39672797ac9638ce4" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 45141 } }
I20260807 07:04:59.554824 10987 raft_consensus.cc:374] T 7645630bc1984deeafb4c5983e6ebfdc P 7882063db22d46a39672797ac9638ce4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:04:59.554916 10987 raft_consensus.cc:729] T 7645630bc1984deeafb4c5983e6ebfdc P 7882063db22d46a39672797ac9638ce4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7882063db22d46a39672797ac9638ce4, State: Initialized, Role: FOLLOWER
I20260807 07:04:59.555397 10987 consensus_queue.cc:260] T 7645630bc1984deeafb4c5983e6ebfdc P 7882063db22d46a39672797ac9638ce4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe73acb9913948e09ae60afc1ebdf24f" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 45093 } } peers { permanent_uuid: "7676cbdb92e040f9b34aef99c0ddbc30" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 45401 } } peers { permanent_uuid: "7882063db22d46a39672797ac9638ce4" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 45141 } }
I20260807 07:04:59.556300 10956 heartbeater.cc:503] Master 127.21.83.62:40691 was elected leader, sending a full tablet report...
I20260807 07:04:59.556849 10987 ts_tablet_manager.cc:1428] T 7645630bc1984deeafb4c5983e6ebfdc P 7882063db22d46a39672797ac9638ce4: Time spent starting tablet: real 0.010s user 0.006s sys 0.004s
W20260807 07:04:59.677600 10957 tablet.cc:2369] T 7645630bc1984deeafb4c5983e6ebfdc P 7882063db22d46a39672797ac9638ce4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 07:04:59.734200 10771 tablet.cc:2369] T 7645630bc1984deeafb4c5983e6ebfdc P fe73acb9913948e09ae60afc1ebdf24f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 07:04:59.783466 10569 tablet.cc:2369] T 7645630bc1984deeafb4c5983e6ebfdc P 7676cbdb92e040f9b34aef99c0ddbc30: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 07:04:59.855029 10993 raft_consensus.cc:482] T 7645630bc1984deeafb4c5983e6ebfdc P fe73acb9913948e09ae60afc1ebdf24f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:04:59.855355 10993 raft_consensus.cc:504] T 7645630bc1984deeafb4c5983e6ebfdc P fe73acb9913948e09ae60afc1ebdf24f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe73acb9913948e09ae60afc1ebdf24f" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 45093 } } peers { permanent_uuid: "7676cbdb92e040f9b34aef99c0ddbc30" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 45401 } } peers { permanent_uuid: "7882063db22d46a39672797ac9638ce4" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 45141 } }
I20260807 07:04:59.857110 10993 leader_election.cc:288] T 7645630bc1984deeafb4c5983e6ebfdc P fe73acb9913948e09ae60afc1ebdf24f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7676cbdb92e040f9b34aef99c0ddbc30 (127.21.83.1:45401), 7882063db22d46a39672797ac9638ce4 (127.21.83.3:45141)
I20260807 07:04:59.864600 10980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:59.857663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:43309 (local address 127.21.83.3:45141)
0807 07:04:59.857880 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:59.857887 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:59.858410 (+ 523us) server_negotiation.cc:407] Connection header received
0807 07:04:59.858629 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:59.858633 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:59.858693 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:59.858767 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:04:59.859420 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:59.860197 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:59.862203 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:59.863366 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:59.863498 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:59.863605 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:59.864147 (+ 542us) server_negotiation.cc:299] Negotiation successful
0807 07:04:59.864317 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 07:04:59.864735 10997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:59.857339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:45401 (local address 127.21.83.2:60103)
0807 07:04:59.858256 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:59.858274 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 07:04:59.858426 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:04:59.858724 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:04:59.858728 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:04:59.858751 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:04:59.859142 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:59.859156 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:59.860175 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:59.860181 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:59.861863 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:59.861878 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:59.863610 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:59.863617 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:59.863866 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:59.863948 (+ 82us) client_negotiation.cc:769] Sending connection context
0807 07:04:59.864045 (+ 97us) client_negotiation.cc:240] Negotiation successful
0807 07:04:59.864251 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"mutex_wait_us":88,"spinlock_wait_cycles":28544,"thread_start_us":538,"threads_started":1}
I20260807 07:04:59.865083 10998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:59.857432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.3:45141 (local address 127.21.83.2:43309)
0807 07:04:59.858300 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0807 07:04:59.858315 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 07:04:59.858495 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:04:59.858794 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:04:59.858798 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:04:59.858825 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:04:59.859248 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:59.859258 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:59.860336 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:59.860340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:59.861991 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:04:59.862006 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:04:59.863650 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:04:59.863657 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:04:59.863945 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:59.863972 (+ 27us) client_negotiation.cc:769] Sending connection context
0807 07:04:59.864045 (+ 73us) client_negotiation.cc:240] Negotiation successful
0807 07:04:59.864251 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"mutex_wait_us":90,"spinlock_wait_cycles":22784,"thread_start_us":560,"threads_started":1}
I20260807 07:04:59.865724 10977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:59.857553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:60103 (local address 127.21.83.1:45401)
0807 07:04:59.857783 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:59.857789 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:59.858371 (+ 582us) server_negotiation.cc:407] Connection header received
0807 07:04:59.858556 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:59.858560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:59.858618 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:59.858693 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:04:59.859282 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:59.860045 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:59.862066 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:59.863335 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:59.863498 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:59.863600 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:59.864858 (+ 1258us) server_negotiation.cc:299] Negotiation successful
0807 07:04:59.865024 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 07:04:59.866115 10888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7645630bc1984deeafb4c5983e6ebfdc" candidate_uuid: "fe73acb9913948e09ae60afc1ebdf24f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7882063db22d46a39672797ac9638ce4" is_pre_election: true
I20260807 07:04:59.866756 10888 raft_consensus.cc:2454] T 7645630bc1984deeafb4c5983e6ebfdc P 7882063db22d46a39672797ac9638ce4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe73acb9913948e09ae60afc1ebdf24f in term 0.
I20260807 07:04:59.866871 10510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7645630bc1984deeafb4c5983e6ebfdc" candidate_uuid: "fe73acb9913948e09ae60afc1ebdf24f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7676cbdb92e040f9b34aef99c0ddbc30" is_pre_election: true
I20260807 07:04:59.867355 10510 raft_consensus.cc:2454] T 7645630bc1984deeafb4c5983e6ebfdc P 7676cbdb92e040f9b34aef99c0ddbc30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe73acb9913948e09ae60afc1ebdf24f in term 0.
I20260807 07:04:59.867513 10596 leader_election.cc:302] T 7645630bc1984deeafb4c5983e6ebfdc P fe73acb9913948e09ae60afc1ebdf24f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7882063db22d46a39672797ac9638ce4, fe73acb9913948e09ae60afc1ebdf24f; no voters:
I20260807 07:04:59.867946 10993 raft_consensus.cc:2781] T 7645630bc1984deeafb4c5983e6ebfdc P fe73acb9913948e09ae60afc1ebdf24f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 07:04:59.868103 10993 raft_consensus.cc:482] T 7645630bc1984deeafb4c5983e6ebfdc P fe73acb9913948e09ae60afc1ebdf24f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:04:59.868193 10993 raft_consensus.cc:3037] T 7645630bc1984deeafb4c5983e6ebfdc P fe73acb9913948e09ae60afc1ebdf24f [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:04:59.870941 10993 raft_consensus.cc:504] T 7645630bc1984deeafb4c5983e6ebfdc P fe73acb9913948e09ae60afc1ebdf24f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe73acb9913948e09ae60afc1ebdf24f" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 45093 } } peers { permanent_uuid: "7676cbdb92e040f9b34aef99c0ddbc30" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 45401 } } peers { permanent_uuid: "7882063db22d46a39672797ac9638ce4" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 45141 } }
I20260807 07:04:59.871608 10993 leader_election.cc:288] T 7645630bc1984deeafb4c5983e6ebfdc P fe73acb9913948e09ae60afc1ebdf24f [CANDIDATE]: Term 1 election: Requested vote from peers 7676cbdb92e040f9b34aef99c0ddbc30 (127.21.83.1:45401), 7882063db22d46a39672797ac9638ce4 (127.21.83.3:45141)
I20260807 07:04:59.872439 10888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7645630bc1984deeafb4c5983e6ebfdc" candidate_uuid: "fe73acb9913948e09ae60afc1ebdf24f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7882063db22d46a39672797ac9638ce4"
I20260807 07:04:59.872484 10510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7645630bc1984deeafb4c5983e6ebfdc" candidate_uuid: "fe73acb9913948e09ae60afc1ebdf24f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7676cbdb92e040f9b34aef99c0ddbc30"
I20260807 07:04:59.872628 10888 raft_consensus.cc:3037] T 7645630bc1984deeafb4c5983e6ebfdc P 7882063db22d46a39672797ac9638ce4 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:04:59.872732 10510 raft_consensus.cc:3037] T 7645630bc1984deeafb4c5983e6ebfdc P 7676cbdb92e040f9b34aef99c0ddbc30 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:04:59.875435 10510 raft_consensus.cc:2454] T 7645630bc1984deeafb4c5983e6ebfdc P 7676cbdb92e040f9b34aef99c0ddbc30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe73acb9913948e09ae60afc1ebdf24f in term 1.
I20260807 07:04:59.875558 10888 raft_consensus.cc:2454] T 7645630bc1984deeafb4c5983e6ebfdc P 7882063db22d46a39672797ac9638ce4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe73acb9913948e09ae60afc1ebdf24f in term 1.
I20260807 07:04:59.875883 10598 leader_election.cc:302] T 7645630bc1984deeafb4c5983e6ebfdc P fe73acb9913948e09ae60afc1ebdf24f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7676cbdb92e040f9b34aef99c0ddbc30, fe73acb9913948e09ae60afc1ebdf24f; no voters:
I20260807 07:04:59.876401 10993 raft_consensus.cc:2781] T 7645630bc1984deeafb4c5983e6ebfdc P fe73acb9913948e09ae60afc1ebdf24f [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:04:59.876868 10993 raft_consensus.cc:686] T 7645630bc1984deeafb4c5983e6ebfdc P fe73acb9913948e09ae60afc1ebdf24f [term 1 LEADER]: Becoming Leader. State: Replica: fe73acb9913948e09ae60afc1ebdf24f, State: Running, Role: LEADER
I20260807 07:04:59.877751 10993 consensus_queue.cc:237] T 7645630bc1984deeafb4c5983e6ebfdc P fe73acb9913948e09ae60afc1ebdf24f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe73acb9913948e09ae60afc1ebdf24f" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 45093 } } peers { permanent_uuid: "7676cbdb92e040f9b34aef99c0ddbc30" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 45401 } } peers { permanent_uuid: "7882063db22d46a39672797ac9638ce4" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 45141 } }
I20260807 07:04:59.882820 10314 catalog_manager.cc:5495] T 7645630bc1984deeafb4c5983e6ebfdc P fe73acb9913948e09ae60afc1ebdf24f reported cstate change: term changed from 0 to 1, leader changed from <none> to fe73acb9913948e09ae60afc1ebdf24f (127.21.83.2). New cstate: current_term: 1 leader_uuid: "fe73acb9913948e09ae60afc1ebdf24f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe73acb9913948e09ae60afc1ebdf24f" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 45093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7676cbdb92e040f9b34aef99c0ddbc30" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 45401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7882063db22d46a39672797ac9638ce4" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 45141 } health_report { overall_health: UNKNOWN } } }
I20260807 07:04:59.951327 10974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:04:59.946408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59988 (local address 127.21.83.2:45093)
0807 07:04:59.946590 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0807 07:04:59.946595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:04:59.946613 (+ 18us) server_negotiation.cc:407] Connection header received
0807 07:04:59.946760 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:04:59.946763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:04:59.946857 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:04:59.946973 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:04:59.947277 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:59.947966 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:59.949418 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:04:59.949835 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:04:59.949927 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:04:59.950363 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:04:59.950904 (+ 541us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:04:59.951013 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0807 07:04:59.951054 (+ 41us) server_negotiation.cc:299] Negotiation successful
0807 07:04:59.951123 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260807 07:04:59.959504 10888 raft_consensus.cc:1261] T 7645630bc1984deeafb4c5983e6ebfdc P 7882063db22d46a39672797ac9638ce4 [term 1 FOLLOWER]: Refusing update from remote peer fe73acb9913948e09ae60afc1ebdf24f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:04:59.959540 10510 raft_consensus.cc:1261] T 7645630bc1984deeafb4c5983e6ebfdc P 7676cbdb92e040f9b34aef99c0ddbc30 [term 1 FOLLOWER]: Refusing update from remote peer fe73acb9913948e09ae60afc1ebdf24f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:04:59.960448 10999 consensus_queue.cc:1035] T 7645630bc1984deeafb4c5983e6ebfdc P fe73acb9913948e09ae60afc1ebdf24f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7882063db22d46a39672797ac9638ce4" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 45141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:04:59.960862 10993 consensus_queue.cc:1035] T 7645630bc1984deeafb4c5983e6ebfdc P fe73acb9913948e09ae60afc1ebdf24f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7676cbdb92e040f9b34aef99c0ddbc30" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 45401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:04:59.978273 11003 mvcc.cc:205] Tried to move back new op lower bound from 7315809484624437248 to 7315809484303413248. Current Snapshot: MvccSnapshot[applied={T|T < 7315809484624437248}]
I20260807 07:04:59.998759 10313 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:50790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:04:59.999034 10313 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:50790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:05:00.002985 10313 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad: Sending DeleteTablet for 3 replicas of tablet 7645630bc1984deeafb4c5983e6ebfdc
I20260807 07:05:00.004298 10667 tablet_service.cc:1514] Processing DeleteTablet for tablet 7645630bc1984deeafb4c5983e6ebfdc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:04:59 UTC) from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:59972
I20260807 07:05:00.004944 10480 tablet_service.cc:1514] Processing DeleteTablet for tablet 7645630bc1984deeafb4c5983e6ebfdc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:04:59 UTC) from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:34400
I20260807 07:05:00.005210 10862 tablet_service.cc:1514] Processing DeleteTablet for tablet 7645630bc1984deeafb4c5983e6ebfdc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:04:59 UTC) from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:41962
I20260807 07:05:00.007233 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 10363
I20260807 07:05:00.016515 11016 tablet_replica.cc:330] T 7645630bc1984deeafb4c5983e6ebfdc P 7676cbdb92e040f9b34aef99c0ddbc30: stopping tablet replica
I20260807 07:05:00.017138 11016 raft_consensus.cc:2229] T 7645630bc1984deeafb4c5983e6ebfdc P 7676cbdb92e040f9b34aef99c0ddbc30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:05:00.017724 11016 raft_consensus.cc:2258] T 7645630bc1984deeafb4c5983e6ebfdc P 7676cbdb92e040f9b34aef99c0ddbc30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:05:00.018625 11015 tablet_replica.cc:330] T 7645630bc1984deeafb4c5983e6ebfdc P fe73acb9913948e09ae60afc1ebdf24f: stopping tablet replica
I20260807 07:05:00.019043 11015 raft_consensus.cc:2229] T 7645630bc1984deeafb4c5983e6ebfdc P fe73acb9913948e09ae60afc1ebdf24f [term 1 LEADER]: Raft consensus shutting down.
I20260807 07:05:00.019872 11015 raft_consensus.cc:2258] T 7645630bc1984deeafb4c5983e6ebfdc P fe73acb9913948e09ae60afc1ebdf24f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:05:00.022068 11015 ts_tablet_manager.cc:1905] T 7645630bc1984deeafb4c5983e6ebfdc P fe73acb9913948e09ae60afc1ebdf24f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:05:00.023695 11016 ts_tablet_manager.cc:1905] T 7645630bc1984deeafb4c5983e6ebfdc P 7676cbdb92e040f9b34aef99c0ddbc30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:05:00.023874 11018 tablet_replica.cc:330] T 7645630bc1984deeafb4c5983e6ebfdc P 7882063db22d46a39672797ac9638ce4: stopping tablet replica
I20260807 07:05:00.024346 11018 raft_consensus.cc:2229] T 7645630bc1984deeafb4c5983e6ebfdc P 7882063db22d46a39672797ac9638ce4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:05:00.024854 11018 raft_consensus.cc:2258] T 7645630bc1984deeafb4c5983e6ebfdc P 7882063db22d46a39672797ac9638ce4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:05:00.029547 11015 ts_tablet_manager.cc:1918] T 7645630bc1984deeafb4c5983e6ebfdc P fe73acb9913948e09ae60afc1ebdf24f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:05:00.029728 11015 log.cc:1196] T 7645630bc1984deeafb4c5983e6ebfdc P fe73acb9913948e09ae60afc1ebdf24f: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-1/wal/wals/7645630bc1984deeafb4c5983e6ebfdc
I20260807 07:05:00.030284 11015 ts_tablet_manager.cc:1939] T 7645630bc1984deeafb4c5983e6ebfdc P fe73acb9913948e09ae60afc1ebdf24f: Deleting consensus metadata
I20260807 07:05:00.032011 10292 catalog_manager.cc:4841] TS fe73acb9913948e09ae60afc1ebdf24f (127.21.83.2:45093): tablet 7645630bc1984deeafb4c5983e6ebfdc (table test-table [id=e3b859aa8ce34e1c98632ddcd134a81e]) successfully deleted
I20260807 07:05:00.032408 11016 ts_tablet_manager.cc:1918] T 7645630bc1984deeafb4c5983e6ebfdc P 7676cbdb92e040f9b34aef99c0ddbc30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:05:00.032543 11016 log.cc:1196] T 7645630bc1984deeafb4c5983e6ebfdc P 7676cbdb92e040f9b34aef99c0ddbc30: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-0/wal/wals/7645630bc1984deeafb4c5983e6ebfdc
I20260807 07:05:00.033099 11016 ts_tablet_manager.cc:1939] T 7645630bc1984deeafb4c5983e6ebfdc P 7676cbdb92e040f9b34aef99c0ddbc30: Deleting consensus metadata
I20260807 07:05:00.034132 11018 ts_tablet_manager.cc:1905] T 7645630bc1984deeafb4c5983e6ebfdc P 7882063db22d46a39672797ac9638ce4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:05:00.038834 10293 catalog_manager.cc:4841] TS 7676cbdb92e040f9b34aef99c0ddbc30 (127.21.83.1:45401): tablet 7645630bc1984deeafb4c5983e6ebfdc (table test-table [id=e3b859aa8ce34e1c98632ddcd134a81e]) successfully deleted
I20260807 07:05:00.040720 11018 ts_tablet_manager.cc:1918] T 7645630bc1984deeafb4c5983e6ebfdc P 7882063db22d46a39672797ac9638ce4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:05:00.040874 11018 log.cc:1196] T 7645630bc1984deeafb4c5983e6ebfdc P 7882063db22d46a39672797ac9638ce4: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-2/wal/wals/7645630bc1984deeafb4c5983e6ebfdc
I20260807 07:05:00.041034 10977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:00.008672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34404 (local address 127.21.83.1:45401)
0807 07:05:00.008875 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0807 07:05:00.008882 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:05:00.021511 (+ 12629us) server_negotiation.cc:407] Connection header received
0807 07:05:00.021721 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:05:00.021726 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:05:00.021825 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:05:00.021934 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:05:00.025633 (+ 3699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:00.026677 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:00.028580 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:00.029240 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:00.029367 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:05:00.033556 (+ 4189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:05:00.033568 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:05:00.033586 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:05:00.033624 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:05:00.037313 (+ 3689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:05:00.038214 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:05:00.038222 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:05:00.038226 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:05:00.038379 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:05:00.039106 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:05:00.039111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:05:00.039115 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:05:00.039580 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:05:00.039950 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0807 07:05:00.040241 (+ 291us) server_negotiation.cc:299] Negotiation successful
0807 07:05:00.040518 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 07:05:00.041414 11018 ts_tablet_manager.cc:1939] T 7645630bc1984deeafb4c5983e6ebfdc P 7882063db22d46a39672797ac9638ce4: Deleting consensus metadata
I20260807 07:05:00.041747 10562 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:05:00.043201 10292 catalog_manager.cc:4841] TS 7882063db22d46a39672797ac9638ce4 (127.21.83.3:45141): tablet 7645630bc1984deeafb4c5983e6ebfdc (table test-table [id=e3b859aa8ce34e1c98632ddcd134a81e]) successfully deleted
I20260807 07:05:00.134683 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 10363
I20260807 07:05:00.158560 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 10575
I20260807 07:05:00.174544 10974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:00.159483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59994 (local address 127.21.83.2:45093)
0807 07:05:00.159689 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0807 07:05:00.159696 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:05:00.159717 (+ 21us) server_negotiation.cc:407] Connection header received
0807 07:05:00.159789 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:05:00.159794 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:05:00.159875 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:05:00.160001 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:05:00.161328 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:00.162603 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:00.163931 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:00.164645 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:00.164778 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:05:00.167406 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:05:00.167426 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:05:00.167447 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:05:00.167491 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:05:00.171418 (+ 3927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:05:00.172142 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:05:00.172151 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:05:00.172155 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:05:00.172331 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:05:00.172657 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:05:00.172662 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:05:00.172665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:05:00.173266 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:05:00.173667 (+ 401us) server_negotiation.cc:1035] Waiting for connection context
0807 07:05:00.173897 (+ 230us) server_negotiation.cc:299] Negotiation successful
0807 07:05:00.174137 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 07:05:00.175426 10762 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:05:00.291358 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 10575
I20260807 07:05:00.315021 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 10773
I20260807 07:05:00.328099 10980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:00.315780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41968 (local address 127.21.83.3:45141)
0807 07:05:00.315957 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0807 07:05:00.315961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:05:00.316025 (+ 64us) server_negotiation.cc:407] Connection header received
0807 07:05:00.316178 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:05:00.316182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:05:00.316247 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:05:00.316332 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:05:00.317871 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:00.318623 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:00.319947 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:00.320340 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:00.320433 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:05:00.323216 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:05:00.323234 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:05:00.323252 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:05:00.323302 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:05:00.325900 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:05:00.326535 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:05:00.326541 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:05:00.326543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:05:00.326670 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:05:00.326938 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:05:00.326941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:05:00.326943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:05:00.327258 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:05:00.327459 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0807 07:05:00.327672 (+ 213us) server_negotiation.cc:299] Negotiation successful
0807 07:05:00.327821 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 07:05:00.328682 10950 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:05:00.417637 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 10773
I20260807 07:05:00.442080 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 10269
I20260807 07:05:00.443439 10355 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:05:00.567312 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 10269
I20260807 07:05:00.593407 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:40691
--webserver_interface=127.21.83.62
--webserver_port=39817
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:37735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:40691
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:05:00.924518 11029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:05:00.924773 11029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:05:00.924913 11029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:05:00.931749 11029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:05:00.931849 11029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:05:00.931924 11029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:05:00.931999 11029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:05:00.932056 11029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:05:00.941618 11029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:37735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:40691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:40691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.83.62
--webserver_port=39817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:05:00.942760 11029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:05:00.943989 11029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:05:00.954909 11035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:05:00.954893 11036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:05:00.954849 11038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:05:00.955926 11029 server_base.cc:1034] running on GCE node
I20260807 07:05:00.956679 11029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:05:00.958020 11029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:05:00.959216 11029 hybrid_clock.cc:648] HybridClock initialized: now 1786086300959160 us; error 46 us; skew 500 ppm
I20260807 07:05:00.959528 11029 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:05:00 dist-test-slave-ncp9 krb5kdc[9502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086300, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:05:00.962524 11029 init.cc:377] Logged in from keytab as kudu/127.21.83.62@KRBTEST.COM (short username kudu)
I20260807 07:05:00.963963 11029 webserver.cc:460] Webserver started at http://127.21.83.62:39817/ using document root <none> and password file <none>
I20260807 07:05:00.964602 11029 fs_manager.cc:359] Metadata directory not provided
I20260807 07:05:00.964718 11029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:05:00.969808 11029 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260807 07:05:00.972342 11056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:05:00.973546 11029 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260807 07:05:00.973702 11029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "189302c9464749e7a638be128648baad"
format_stamp: "Formatted at 2026-08-07 07:04:54 on dist-test-slave-ncp9"
server_key: "8361948d5d4132a9cd9b764014e18406"
server_key_iv: "e35ee20c7aec6fa64cede26b417bf78d"
server_key_version: "encryptionkey@0"
I20260807 07:05:00.974607 11029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:05:01.003413 11029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:05:01.008020 11029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:05:01.008303 11029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:05:01.026819 11029 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.62:40691
I20260807 07:05:01.026844 11143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.62:40691 every 8 connection(s)
I20260807 07:05:01.028221 11029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:05:01.031141 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 11029
I20260807 07:05:01.032302 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.1:45401
--local_ip_for_outbound_sockets=127.21.83.1
--tserver_master_addrs=127.21.83.62:40691
--webserver_port=38133
--webserver_interface=127.21.83.1
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab
--principal=kudu/127.21.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:37735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 07:05:01.036446 11144 sys_catalog.cc:263] Verifying existing consensus state
I20260807 07:05:01.040675 11144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad: Bootstrap starting.
I20260807 07:05:01.062139 11144 log.cc:824] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad: Log is configured to *not* fsync() on all Append() calls
I20260807 07:05:01.078380 11144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 07:05:01.078894 11144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad: Bootstrap complete.
I20260807 07:05:01.087931 11144 raft_consensus.cc:348] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "189302c9464749e7a638be128648baad" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 40691 } }
I20260807 07:05:01.088814 11144 raft_consensus.cc:729] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 189302c9464749e7a638be128648baad, State: Initialized, Role: FOLLOWER
I20260807 07:05:01.089390 11144 consensus_queue.cc:260] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "189302c9464749e7a638be128648baad" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 40691 } }
I20260807 07:05:01.089599 11144 raft_consensus.cc:388] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:05:01.089720 11144 raft_consensus.cc:482] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:05:01.089869 11144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad [term 2 FOLLOWER]: Advancing to term 3
I20260807 07:05:01.093497 11144 raft_consensus.cc:504] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "189302c9464749e7a638be128648baad" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 40691 } }
I20260807 07:05:01.093932 11144 leader_election.cc:302] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 189302c9464749e7a638be128648baad; no voters:
I20260807 07:05:01.094507 11144 leader_election.cc:288] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad [CANDIDATE]: Term 3 election: Requested vote from peers
I20260807 07:05:01.094645 11151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad [term 3 FOLLOWER]: Leader election won for term 3
I20260807 07:05:01.095733 11151 raft_consensus.cc:686] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad [term 3 LEADER]: Becoming Leader. State: Replica: 189302c9464749e7a638be128648baad, State: Running, Role: LEADER
I20260807 07:05:01.096341 11151 consensus_queue.cc:237] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "189302c9464749e7a638be128648baad" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 40691 } }
I20260807 07:05:01.096529 11144 sys_catalog.cc:564] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:05:01.098820 11154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 189302c9464749e7a638be128648baad. Latest consensus state: current_term: 3 leader_uuid: "189302c9464749e7a638be128648baad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "189302c9464749e7a638be128648baad" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 40691 } } }
I20260807 07:05:01.099546 11154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad [sys.catalog]: This master's current role is: LEADER
I20260807 07:05:01.099217 11153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "189302c9464749e7a638be128648baad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "189302c9464749e7a638be128648baad" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 40691 } } }
I20260807 07:05:01.100041 11153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 189302c9464749e7a638be128648baad [sys.catalog]: This master's current role is: LEADER
I20260807 07:05:01.107236 11160 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:05:01.115115 11160 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:05:01.116529 11160 catalog_manager.cc:1259] Loaded cluster ID: f014a38f54fe49d19698d571a1c22ed8
I20260807 07:05:01.116623 11160 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260807 07:05:01.118670 11160 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f1a52d259ad google::LogMessage::Fail() at ??:0
@ 0x7f1a52d2888a google::LogMessage::SendToLog() at ??:0
@ 0x7f1a52d254de google::LogMessage::Flush() at ??:0
@ 0x7f1a52d26a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f1a68758512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f1a68755976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f1a687dd610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f1a687dd439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f1a55971345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1a5597491c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f1a55974799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f1a5594feaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1a58c296db start_thread at ??:0
@ 0x7f1a51bbf61f clone at ??:0
W20260807 07:05:01.437181 11148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:05:01.437505 11148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:05:01.437656 11148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:05:01.446910 11148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:05:01.447070 11148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:05:01.447255 11148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.1
I20260807 07:05:01.458528 11148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:37735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab
--principal=kudu/127.21.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.1:45401
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.83.1
--webserver_port=38133
--tserver_master_addrs=127.21.83.62:40691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.1
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:05:01.459810 11148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:05:01.461115 11148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:05:01.473742 11198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:05:01.473706 11201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:05:01.473716 11199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:05:01.474797 11148 server_base.cc:1034] running on GCE node
I20260807 07:05:01.475364 11148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:05:01.476229 11148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:05:01.477423 11148 hybrid_clock.cc:648] HybridClock initialized: now 1786086301477383 us; error 30 us; skew 500 ppm
I20260807 07:05:01.477747 11148 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:05:01 dist-test-slave-ncp9 krb5kdc[9502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086301, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:05:01.480634 11148 init.cc:377] Logged in from keytab as kudu/127.21.83.1@KRBTEST.COM (short username kudu)
I20260807 07:05:01.482010 11148 webserver.cc:460] Webserver started at http://127.21.83.1:38133/ using document root <none> and password file <none>
I20260807 07:05:01.482678 11148 fs_manager.cc:359] Metadata directory not provided
I20260807 07:05:01.482798 11148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:05:01.488025 11148 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20260807 07:05:01.490664 11212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:05:01.491967 11148 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 07:05:01.492142 11148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-0/wal
uuid: "7676cbdb92e040f9b34aef99c0ddbc30"
format_stamp: "Formatted at 2026-08-07 07:04:55 on dist-test-slave-ncp9"
server_key: "c2205ded58806d934c6bd252549406ab"
server_key_iv: "d5f9ea3655889aff94521f9360313c0d"
server_key_version: "encryptionkey@0"
I20260807 07:05:01.493191 11148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:05:01.512466 11148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:05:01.516718 11148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:05:01.517019 11148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:05:01.518081 11148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:05:01.519657 11148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:05:01.519783 11148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:05:01.519897 11148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:05:01.519970 11148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:05:01.551029 11148 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.1:45401
I20260807 07:05:01.551095 11375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.1:45401 every 8 connection(s)
I20260807 07:05:01.552416 11148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
I20260807 07:05:01.556147 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 11148
I20260807 07:05:01.556792 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.2:45093
--local_ip_for_outbound_sockets=127.21.83.2
--tserver_master_addrs=127.21.83.62:40691
--webserver_port=36915
--webserver_interface=127.21.83.2
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab
--principal=kudu/127.21.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:37735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 07:05:01.556841 11382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:01.555044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.1:52513)
0807 07:05:01.555787 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:01.555970 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":290,"threads_started":1}
W20260807 07:05:01.557539 11376 heartbeater.cc:644] Failed to heartbeat to 127.21.83.62:40691 (0 consecutive failures): Network error: Failed to ping master at 127.21.83.62:40691: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
I20260807 07:05:01.558414 11382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:01.558031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.1:57425)
0807 07:05:01.558195 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:01.558277 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260807 07:05:01.559234 11382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:01.558880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.1:38435)
0807 07:05:01.559011 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:01.559098 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260807 07:05:01.559450 11376 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 07:05:01.895632 11383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:05:01.895893 11383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:05:01.896015 11383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:05:01.903192 11383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:05:01.903293 11383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:05:01.903466 11383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.2
I20260807 07:05:01.912693 11383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:37735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab
--principal=kudu/127.21.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.2:45093
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.83.2
--webserver_port=36915
--tserver_master_addrs=127.21.83.62:40691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.2
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:05:01.913806 11383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:05:01.914985 11383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:05:01.930166 11392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:05:01.930207 11389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:05:01.930158 11390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:05:01.931135 11383 server_base.cc:1034] running on GCE node
I20260807 07:05:01.931835 11383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:05:01.932919 11383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:05:01.934157 11383 hybrid_clock.cc:648] HybridClock initialized: now 1786086301934102 us; error 44 us; skew 500 ppm
I20260807 07:05:01.934491 11383 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:05:01 dist-test-slave-ncp9 krb5kdc[9502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086301, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:05:01.937752 11383 init.cc:377] Logged in from keytab as kudu/127.21.83.2@KRBTEST.COM (short username kudu)
I20260807 07:05:01.939371 11383 webserver.cc:460] Webserver started at http://127.21.83.2:36915/ using document root <none> and password file <none>
I20260807 07:05:01.940119 11383 fs_manager.cc:359] Metadata directory not provided
I20260807 07:05:01.940250 11383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:05:01.946031 11383 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260807 07:05:01.948792 11400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:05:01.950147 11383 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 07:05:01.950309 11383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-1/wal
uuid: "fe73acb9913948e09ae60afc1ebdf24f"
format_stamp: "Formatted at 2026-08-07 07:04:55 on dist-test-slave-ncp9"
server_key: "10d60cbdc41369e1502319772e11b105"
server_key_iv: "a99087ea8866ef68b76a9cf1039ce550"
server_key_version: "encryptionkey@0"
I20260807 07:05:01.951306 11383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:05:01.974359 11383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:05:01.978379 11383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:05:01.978637 11383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:05:01.979628 11383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:05:01.980986 11383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:05:01.981094 11383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:05:01.981209 11383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:05:01.981280 11383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:05:02.014284 11383 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.2:45093
I20260807 07:05:02.014358 11568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.2:45093 every 8 connection(s)
I20260807 07:05:02.015784 11383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
I20260807 07:05:02.016270 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 11383
I20260807 07:05:02.017048 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.3:45141
--local_ip_for_outbound_sockets=127.21.83.3
--tserver_master_addrs=127.21.83.62:40691
--webserver_port=33465
--webserver_interface=127.21.83.3
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab
--principal=kudu/127.21.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:37735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 07:05:02.021294 11573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:02.018948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.2:45449)
0807 07:05:02.019840 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:02.020097 (+ 257us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":346,"threads_started":1}
W20260807 07:05:02.022030 11570 heartbeater.cc:644] Failed to heartbeat to 127.21.83.62:40691 (0 consecutive failures): Network error: Failed to ping master at 127.21.83.62:40691: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
I20260807 07:05:02.022918 11573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:02.022536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.2:46573)
0807 07:05:02.022706 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:02.022791 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260807 07:05:02.023819 11573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:02.023451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.2:37059)
0807 07:05:02.023592 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:02.023693 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260807 07:05:02.024076 11570 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 07:05:02.347510 11574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:05:02.347765 11574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:05:02.347896 11574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:05:02.354781 11574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:05:02.354887 11574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:05:02.355037 11574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.3
I20260807 07:05:02.364234 11574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:37735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab
--principal=kudu/127.21.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.3:45141
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.83.3
--webserver_port=33465
--tserver_master_addrs=127.21.83.62:40691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.3
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:05:02.365389 11574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:05:02.366603 11574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:05:02.378566 11583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:05:02.378669 11585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:05:02.378763 11587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:05:02.379640 11574 server_base.cc:1034] running on GCE node
I20260807 07:05:02.380167 11574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:05:02.381029 11574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:05:02.382261 11574 hybrid_clock.cc:648] HybridClock initialized: now 1786086302382200 us; error 49 us; skew 500 ppm
I20260807 07:05:02.382551 11574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:05:02 dist-test-slave-ncp9 krb5kdc[9502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086302, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:05:02.385447 11574 init.cc:377] Logged in from keytab as kudu/127.21.83.3@KRBTEST.COM (short username kudu)
I20260807 07:05:02.386770 11574 webserver.cc:460] Webserver started at http://127.21.83.3:33465/ using document root <none> and password file <none>
I20260807 07:05:02.387344 11574 fs_manager.cc:359] Metadata directory not provided
I20260807 07:05:02.387444 11574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:05:02.392241 11574 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260807 07:05:02.394755 11594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:05:02.395917 11574 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260807 07:05:02.396061 11574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-2/wal
uuid: "7882063db22d46a39672797ac9638ce4"
format_stamp: "Formatted at 2026-08-07 07:04:56 on dist-test-slave-ncp9"
server_key: "9a7eedc45af41856860fbb6aef7e54ba"
server_key_iv: "bb8699a76a13c87918cb06bbfc46f5f9"
server_key_version: "encryptionkey@0"
I20260807 07:05:02.397033 11574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:05:02.419525 11574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:05:02.423414 11574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:05:02.423678 11574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:05:02.424674 11574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:05:02.426002 11574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:05:02.426092 11574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:05:02.426204 11574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:05:02.426271 11574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:05:02.456014 11574 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.3:45141
I20260807 07:05:02.456089 11744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.3:45141 every 8 connection(s)
I20260807 07:05:02.457469 11574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
I20260807 07:05:02.461920 11751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:02.460193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.3:41811)
0807 07:05:02.460870 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:02.461104 (+ 234us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":265,"threads_started":1}
W20260807 07:05:02.462525 11748 heartbeater.cc:644] Failed to heartbeat to 127.21.83.62:40691 (0 consecutive failures): Network error: Failed to ping master at 127.21.83.62:40691: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
I20260807 07:05:02.463217 11751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:02.462936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.3:39581)
0807 07:05:02.463046 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:02.463109 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260807 07:05:02.463891 11751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:02.463635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.3:53399)
0807 07:05:02.463731 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:02.463797 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260807 07:05:02.464078 11748 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260807 07:05:02.466058 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 11574
I20260807 07:05:02.560796 11755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:02.559996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.1:36243)
0807 07:05:02.560435 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:02.560549 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":213,"threads_started":1}
I20260807 07:05:03.025388 11757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:03.024570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.2:49549)
0807 07:05:03.025059 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:03.025165 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":222,"threads_started":1}
I20260807 07:05:03.465349 11758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:03.464537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.3:37269)
0807 07:05:03.465038 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:03.465138 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":265,"threads_started":1}
I20260807 07:05:03.562358 11759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:03.561656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.1:54517)
0807 07:05:03.562104 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:03.562181 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":225,"threads_started":1}
I20260807 07:05:04.026925 11760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:04.026210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.2:44071)
0807 07:05:04.026679 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:04.026760 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":230,"threads_started":1}
I20260807 07:05:04.466794 11761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:04.466119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.3:37807)
0807 07:05:04.466557 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:04.466635 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":213,"threads_started":1}
I20260807 07:05:04.563853 11762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:04.563162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.1:57435)
0807 07:05:04.563597 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:04.563675 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":231,"threads_started":1}
I20260807 07:05:05.028467 11766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:05.027745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.2:49555)
0807 07:05:05.028206 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:05.028290 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":250,"threads_started":1}
I20260807 07:05:05.468317 11768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:05.467618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.3:58681)
0807 07:05:05.468063 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:05.468149 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":242,"threads_started":1}
I20260807 07:05:05.565322 11770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:05.564624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.1:36871)
0807 07:05:05.565062 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:05.565138 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":190,"threads_started":1}
I20260807 07:05:06.030123 11771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:06.029374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.2:46845)
0807 07:05:06.029835 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:06.029916 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":222,"threads_started":1}
I20260807 07:05:06.469830 11772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:06.469156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.3:57247)
0807 07:05:06.469570 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:06.469653 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":209,"threads_started":1}
I20260807 07:05:06.566805 11774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:06.566107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.1:39637)
0807 07:05:06.566549 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:06.566634 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":231,"threads_started":1}
I20260807 07:05:07.031668 11775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:07.030928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.2:53307)
0807 07:05:07.031423 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:07.031501 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":273,"threads_started":1}
I20260807 07:05:07.471378 11776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:07.470691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.3:52687)
0807 07:05:07.471128 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:07.471209 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":217,"threads_started":1}
I20260807 07:05:07.568276 11777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:07.567573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.1:36753)
0807 07:05:07.568026 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:07.568115 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":232,"threads_started":1}
I20260807 07:05:08.033389 11778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:08.032614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.2:41945)
0807 07:05:08.033122 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:08.033218 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":268,"threads_started":1}
I20260807 07:05:08.472935 11779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:08.472225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.3:52227)
0807 07:05:08.472660 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:08.472747 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":224,"threads_started":1}
I20260807 07:05:08.569800 11781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:08.569054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.1:37255)
0807 07:05:08.569515 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:08.569594 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":204,"threads_started":1}
I20260807 07:05:09.035085 11783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:09.034274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.2:59867)
0807 07:05:09.034773 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:09.034853 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":223,"threads_started":1}
I20260807 07:05:09.474651 11785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:09.473856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.3:53333)
0807 07:05:09.474348 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:09.474431 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":216,"threads_started":1}
I20260807 07:05:09.571260 11786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:09.570594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.1:48271)
0807 07:05:09.571008 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:09.571094 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":214,"threads_started":1}
I20260807 07:05:10.036810 11788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:10.036023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.2:56761)
0807 07:05:10.036546 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:10.036636 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":263,"threads_started":1}
I20260807 07:05:10.476114 11789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:10.475470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.3:37701)
0807 07:05:10.475881 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:10.475957 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":217,"threads_started":1}
I20260807 07:05:10.572791 11790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:10.572075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.1:57815)
0807 07:05:10.572508 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:10.572588 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":204,"threads_started":1}
I20260807 07:05:11.038386 11791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:11.037669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.2:36833)
0807 07:05:11.038129 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:11.038211 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":219,"threads_started":1}
I20260807 07:05:11.477979 11792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:11.477106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.3:50549)
0807 07:05:11.477672 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:11.477780 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":269,"threads_started":1}
I20260807 07:05:11.574723 11793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:11.573894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.1:56727)
0807 07:05:11.574428 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:11.574524 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":298,"threads_started":1}
I20260807 07:05:12.040352 11794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:12.039426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.2:40793)
0807 07:05:12.040094 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:12.040181 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":433,"threads_started":1}
I20260807 07:05:12.479983 11795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:12.479133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.3:55057)
0807 07:05:12.479712 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:12.479809 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":307,"threads_started":1}
I20260807 07:05:12.576478 11798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:12.575686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.1:49055)
0807 07:05:12.576177 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:12.576277 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":243,"threads_started":1}
I20260807 07:05:13.042028 11799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:13.041223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.2:46987)
0807 07:05:13.041745 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:13.041845 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":269,"threads_started":1}
I20260807 07:05:13.481950 11800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:13.481134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.3:34967)
0807 07:05:13.481662 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:13.481769 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":266,"threads_started":1}
I20260807 07:05:13.578333 11801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:13.577562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.1:51959)
0807 07:05:13.578064 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:13.578161 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":241,"threads_started":1}
I20260807 07:05:14.043797 11803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:14.043004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.2:49901)
0807 07:05:14.043530 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:14.043625 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":254,"threads_started":1}
I20260807 07:05:14.483791 11804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:14.483017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.3:39475)
0807 07:05:14.483509 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:14.483619 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":240,"threads_started":1}
I20260807 07:05:14.579968 11806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:14.579188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.1:36583)
0807 07:05:14.579666 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:14.579757 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":238,"threads_started":1}
I20260807 07:05:15.045590 11807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:15.044787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.2:41577)
0807 07:05:15.045297 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:15.045391 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":263,"threads_started":1}
I20260807 07:05:15.485550 11808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:15.484744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.3:48331)
0807 07:05:15.485261 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:15.485355 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":270,"threads_started":1}
I20260807 07:05:15.581501 11809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:15.580786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.1:44175)
0807 07:05:15.581237 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:15.581318 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":240,"threads_started":1}
I20260807 07:05:16.047276 11810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:16.046438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.2:44023)
0807 07:05:16.046924 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:16.047022 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":263,"threads_started":1}
I20260807 07:05:16.487443 11811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:16.486628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.3:40027)
0807 07:05:16.487156 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:16.487250 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":272,"threads_started":1}
I20260807 07:05:16.583447 11812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:16.582683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.1:52585)
0807 07:05:16.583136 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:16.583246 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":223,"threads_started":1}
I20260807 07:05:17.049093 11813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:17.048285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.2:38685)
0807 07:05:17.048760 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:17.048875 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":249,"threads_started":1}
I20260807 07:05:17.469221 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 11148
I20260807 07:05:17.487534 11814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:17.470972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47100 (local address 127.21.83.1:45401)
0807 07:05:17.471748 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0807 07:05:17.471767 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0807 07:05:17.471812 (+ 45us) server_negotiation.cc:407] Connection header received
0807 07:05:17.473335 (+ 1523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:05:17.473345 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:05:17.473746 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:05:17.474027 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:05:17.475411 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:17.476680 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:17.478039 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:17.478724 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:17.478862 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:05:17.481538 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:05:17.481546 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:05:17.481561 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:05:17.481603 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:05:17.484490 (+ 2887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:05:17.485248 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:05:17.485256 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:05:17.485262 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:05:17.485457 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:05:17.485777 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:05:17.485780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:05:17.485782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:05:17.486313 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:05:17.486548 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0807 07:05:17.486906 (+ 358us) server_negotiation.cc:299] Negotiation successful
0807 07:05:17.487250 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":446,"threads_started":1}
I20260807 07:05:17.489018 11370 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:05:17.489326 11815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:17.488534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:40691 (local address 127.21.83.3:60355)
0807 07:05:17.489022 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:17.489151 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":247,"threads_started":1}
I20260807 07:05:17.572584 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 11148
I20260807 07:05:17.596230 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 11383
I20260807 07:05:17.614493 11818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:17.597167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56722 (local address 127.21.83.2:45093)
0807 07:05:17.597653 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0807 07:05:17.597668 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 07:05:17.597704 (+ 36us) server_negotiation.cc:407] Connection header received
0807 07:05:17.599144 (+ 1440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:05:17.599154 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:05:17.599595 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:05:17.599925 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:05:17.601706 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:17.603031 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:17.604383 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:17.605189 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:17.605372 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:05:17.608082 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:05:17.608092 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:05:17.608109 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:05:17.608153 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:05:17.611053 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:05:17.611788 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:05:17.611796 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:05:17.611803 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:05:17.611988 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:05:17.612370 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:05:17.612374 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:05:17.612376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:05:17.612983 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:05:17.613292 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0807 07:05:17.613761 (+ 469us) server_negotiation.cc:299] Negotiation successful
0807 07:05:17.614200 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":214,"threads_started":1}
I20260807 07:05:17.616019 11561 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:05:17.700852 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 11383
I20260807 07:05:17.731899 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 11574
I20260807 07:05:17.748704 11820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:17.733049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34118 (local address 127.21.83.3:45141)
0807 07:05:17.733680 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0807 07:05:17.733694 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 07:05:17.733734 (+ 40us) server_negotiation.cc:407] Connection header received
0807 07:05:17.735328 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:05:17.735337 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:05:17.735752 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:05:17.736022 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:05:17.737393 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:17.738590 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:17.739765 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:17.740439 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:17.740588 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:05:17.743136 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:05:17.743143 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:05:17.743157 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:05:17.743198 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:05:17.745906 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:05:17.746520 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:05:17.746526 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:05:17.746531 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:05:17.746697 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:05:17.747039 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:05:17.747043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:05:17.747046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:05:17.747562 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:05:17.747810 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0807 07:05:17.748095 (+ 285us) server_negotiation.cc:299] Negotiation successful
0807 07:05:17.748392 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":279,"threads_started":1}
I20260807 07:05:17.749979 11738 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:05:17.831446 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 11574
2026-08-07T07:05:17Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25145 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 07:05:17 dist-test-slave-ncp9 krb5kdc[11828](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 07:05:17 dist-test-slave-ncp9 krb5kdc[11828](info): set up 2 sockets
Aug 07 07:05:17 dist-test-slave-ncp9 krb5kdc[11828](info): commencing operation
krb5kdc: starting...
W20260807 07:05:19.066104 21836 mini_kdc.cc:121] Time spent starting KDC: real 1.184s user 0.000s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Aug 07 07:05:19 dist-test-slave-ncp9 krb5kdc[11828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T07:05:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:05:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
I20260807 07:05:19.374552 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:39001
--webserver_interface=127.21.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:38161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:39001
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:05:19.710764 11848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:05:19.711145 11848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:05:19.711256 11848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:05:19.719063 11848 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260807 07:05:19.719189 11848 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260807 07:05:19.719228 11848 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260807 07:05:19.719295 11848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:05:19.719349 11848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:05:19.719426 11848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:05:19.719470 11848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:05:19.719527 11848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:05:19.730146 11848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:38161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:39001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:39001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:05:19.731457 11848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:05:19.732810 11848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:05:19.745263 11865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:05:19.745280 11859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:05:19.745374 11857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:05:19.745906 11848 server_base.cc:1034] running on GCE node
I20260807 07:05:19.746801 11848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:05:19.748129 11848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:05:19.749339 11848 hybrid_clock.cc:648] HybridClock initialized: now 1786086319749273 us; error 48 us; skew 500 ppm
I20260807 07:05:19.749660 11848 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:05:19 dist-test-slave-ncp9 krb5kdc[11828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086319, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:05:19.752652 11848 init.cc:377] Logged in from keytab as kudu/127.21.83.62@KRBTEST.COM (short username kudu)
I20260807 07:05:19.754083 11848 webserver.cc:460] Webserver started at http://127.21.83.62:46785/ using document root <none> and password file <none>
I20260807 07:05:19.754756 11848 fs_manager.cc:359] Metadata directory not provided
I20260807 07:05:19.754854 11848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:05:19.755144 11848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:05:19.758057 11848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "dfc9b269350e44038e90a41e5b96c1f5"
format_stamp: "Formatted at 2026-08-07 07:05:19 on dist-test-slave-ncp9"
server_key: "229338d6ef901aca7a97ec9540716f87"
server_key_iv: "66fbb33c791cd5224cd12e9d5f734f93"
server_key_version: "encryptionkey@0"
I20260807 07:05:19.758800 11848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "dfc9b269350e44038e90a41e5b96c1f5"
format_stamp: "Formatted at 2026-08-07 07:05:19 on dist-test-slave-ncp9"
server_key: "229338d6ef901aca7a97ec9540716f87"
server_key_iv: "66fbb33c791cd5224cd12e9d5f734f93"
server_key_version: "encryptionkey@0"
I20260807 07:05:19.763761 11848 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260807 07:05:19.767171 11876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:05:19.768424 11848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 07:05:19.768590 11848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "dfc9b269350e44038e90a41e5b96c1f5"
format_stamp: "Formatted at 2026-08-07 07:05:19 on dist-test-slave-ncp9"
server_key: "229338d6ef901aca7a97ec9540716f87"
server_key_iv: "66fbb33c791cd5224cd12e9d5f734f93"
server_key_version: "encryptionkey@0"
I20260807 07:05:19.768796 11848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:05:19.796556 11848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:05:19.800824 11848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:05:19.801119 11848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:05:19.819043 11848 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.62:39001
I20260807 07:05:19.819069 11953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.62:39001 every 8 connection(s)
I20260807 07:05:19.820325 11848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:05:19.823921 11954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:05:19.826328 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 11848
I20260807 07:05:19.826545 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:05:19.827200 21836 external_mini_cluster.cc:1428] Setting key 08b912fcc5ba30e050bdc6bf6a5b45ad
I20260807 07:05:19.837618 11954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5: Bootstrap starting.
I20260807 07:05:19.841491 11954 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5: Neither blocks nor log segments found. Creating new log.
I20260807 07:05:19.842635 11954 log.cc:824] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5: Log is configured to *not* fsync() on all Append() calls
Aug 07 07:05:19 dist-test-slave-ncp9 krb5kdc[11828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:05:19.846850 11954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5: No bootstrap required, opened a new log
I20260807 07:05:19.856519 11959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:19.829602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48612 (local address 127.21.83.62:39001)
0807 07:05:19.830500 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0807 07:05:19.830516 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 07:05:19.830567 (+ 51us) server_negotiation.cc:407] Connection header received
0807 07:05:19.833051 (+ 2484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:05:19.833086 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:05:19.833703 (+ 617us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:05:19.834214 (+ 511us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:05:19.835782 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:19.837845 (+ 2063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:19.839357 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:19.840456 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:19.840666 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:05:19.846558 (+ 5892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:05:19.846571 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:05:19.846593 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:05:19.846652 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:05:19.851002 (+ 4350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:05:19.851975 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:05:19.851985 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:05:19.851996 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:05:19.852257 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:05:19.852750 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:05:19.852758 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:05:19.852762 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:05:19.853622 (+ 860us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:05:19.853957 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0807 07:05:19.854391 (+ 434us) server_negotiation.cc:299] Negotiation successful
0807 07:05:19.854933 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":390,"threads_started":1}
I20260807 07:05:19.857810 11954 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc9b269350e44038e90a41e5b96c1f5" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39001 } }
I20260807 07:05:19.858201 11954 raft_consensus.cc:374] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:05:19.858307 11954 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfc9b269350e44038e90a41e5b96c1f5, State: Initialized, Role: FOLLOWER
I20260807 07:05:19.858901 11954 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc9b269350e44038e90a41e5b96c1f5" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39001 } }
I20260807 07:05:19.859141 11954 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:05:19.859275 11954 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:05:19.859434 11954 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:05:19.862027 11954 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc9b269350e44038e90a41e5b96c1f5" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39001 } }
I20260807 07:05:19.862610 11954 leader_election.cc:302] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfc9b269350e44038e90a41e5b96c1f5; no voters:
I20260807 07:05:19.863248 11954 leader_election.cc:288] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 07:05:19.863387 11961 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:05:19.864141 11961 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5 [term 1 LEADER]: Becoming Leader. State: Replica: dfc9b269350e44038e90a41e5b96c1f5, State: Running, Role: LEADER
I20260807 07:05:19.864826 11961 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc9b269350e44038e90a41e5b96c1f5" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39001 } }
I20260807 07:05:19.865389 11954 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:05:19.867942 11963 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dfc9b269350e44038e90a41e5b96c1f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc9b269350e44038e90a41e5b96c1f5" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39001 } } }
I20260807 07:05:19.868278 11963 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5 [sys.catalog]: This master's current role is: LEADER
I20260807 07:05:19.868680 11964 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfc9b269350e44038e90a41e5b96c1f5. Latest consensus state: current_term: 1 leader_uuid: "dfc9b269350e44038e90a41e5b96c1f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc9b269350e44038e90a41e5b96c1f5" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39001 } } }
I20260807 07:05:19.869061 11964 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5 [sys.catalog]: This master's current role is: LEADER
I20260807 07:05:19.873023 11967 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:05:19.878623 11967 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:05:19.887503 11967 catalog_manager.cc:1347] Generated new cluster ID: 0f5b480469e24dcfbc8546b45d643a8d
I20260807 07:05:19.887652 11967 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:05:19.916312 11967 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:05:19.918196 11967 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:05:19.927480 11967 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5: Generated new TSK 0
I20260807 07:05:19.928432 11967 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.83.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab.
Entry for principal kudu/127.21.83.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab.
I20260807 07:05:19.998504 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.1:0
--local_ip_for_outbound_sockets=127.21.83.1
--webserver_interface=127.21.83.1
--webserver_port=0
--tserver_master_addrs=127.21.83.62:39001
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab
--principal=kudu/127.21.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:38161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:05:20.343008 11993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:05:20.343355 11993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:05:20.343510 11993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:05:20.351087 11993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:05:20.351230 11993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:05:20.351395 11993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.1
I20260807 07:05:20.361610 11993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:38161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab
--principal=kudu/127.21.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.83.1
--webserver_port=0
--tserver_master_addrs=127.21.83.62:39001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.1
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:05:20.362951 11993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:05:20.364249 11993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:05:20.377141 12002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:05:20.377202 12006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:05:20.377096 12003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:05:20.378137 11993 server_base.cc:1034] running on GCE node
I20260807 07:05:20.378741 11993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:05:20.379560 11993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:05:20.380761 11993 hybrid_clock.cc:648] HybridClock initialized: now 1786086320380723 us; error 26 us; skew 500 ppm
I20260807 07:05:20.381093 11993 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:05:20 dist-test-slave-ncp9 krb5kdc[11828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086320, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:05:20.387792 11993 init.cc:377] Logged in from keytab as kudu/127.21.83.1@KRBTEST.COM (short username kudu)
I20260807 07:05:20.389108 11993 webserver.cc:460] Webserver started at http://127.21.83.1:34403/ using document root <none> and password file <none>
I20260807 07:05:20.389744 11993 fs_manager.cc:359] Metadata directory not provided
I20260807 07:05:20.389863 11993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:05:20.390146 11993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:05:20.392940 11993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-0/data/instance:
uuid: "faefba3fc057402dae24f9e78dfdf048"
format_stamp: "Formatted at 2026-08-07 07:05:20 on dist-test-slave-ncp9"
server_key: "70ce1b31c472ebdce137a2a6cf576add"
server_key_iv: "c907f32db2236ebaf31cab5bcc6d84c3"
server_key_version: "encryptionkey@0"
I20260807 07:05:20.393682 11993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-0/wal/instance:
uuid: "faefba3fc057402dae24f9e78dfdf048"
format_stamp: "Formatted at 2026-08-07 07:05:20 on dist-test-slave-ncp9"
server_key: "70ce1b31c472ebdce137a2a6cf576add"
server_key_iv: "c907f32db2236ebaf31cab5bcc6d84c3"
server_key_version: "encryptionkey@0"
I20260807 07:05:20.398527 11993 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260807 07:05:20.401886 12018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:05:20.403044 11993 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260807 07:05:20.403201 11993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-0/wal
uuid: "faefba3fc057402dae24f9e78dfdf048"
format_stamp: "Formatted at 2026-08-07 07:05:20 on dist-test-slave-ncp9"
server_key: "70ce1b31c472ebdce137a2a6cf576add"
server_key_iv: "c907f32db2236ebaf31cab5bcc6d84c3"
server_key_version: "encryptionkey@0"
I20260807 07:05:20.403371 11993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:05:20.428679 11993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:05:20.432449 11993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:05:20.432713 11993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:05:20.433732 11993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:05:20.435091 11993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:05:20.435180 11993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:05:20.435302 11993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:05:20.435354 11993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:05:20.468122 11993 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.1:32771
I20260807 07:05:20.468187 12164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.1:32771 every 8 connection(s)
I20260807 07:05:20.469528 11993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
I20260807 07:05:20.470388 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 11993
I20260807 07:05:20.470595 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-0/wal/instance
I20260807 07:05:20.471416 21836 external_mini_cluster.cc:1428] Setting key 5ae4311bee58c1f6cb1d888ce57d40f7
Aug 07 07:05:20 dist-test-slave-ncp9 krb5kdc[11828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086320, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.1@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:05:20.492486 12167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:20.473064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:56865 (local address 127.21.83.62:39001)
0807 07:05:20.473591 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0807 07:05:20.473597 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:05:20.474836 (+ 1239us) server_negotiation.cc:407] Connection header received
0807 07:05:20.476814 (+ 1978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:05:20.476821 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:05:20.476934 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:05:20.477049 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:05:20.478687 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:20.479656 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:20.481405 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:20.481824 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:20.481954 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:05:20.485770 (+ 3816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:05:20.485776 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:05:20.485779 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:05:20.485819 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:05:20.488918 (+ 3099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:05:20.489893 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:05:20.489899 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:05:20.489903 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:05:20.490072 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:05:20.490531 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:05:20.490537 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:05:20.490541 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:05:20.490891 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:05:20.491037 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0807 07:05:20.491857 (+ 820us) server_negotiation.cc:299] Negotiation successful
0807 07:05:20.492125 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":272,"threads_started":1}
I20260807 07:05:20.493394 12168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:20.473738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.1:56865)
0807 07:05:20.474646 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:20.474695 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 07:05:20.476512 (+ 1817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:05:20.477261 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:05:20.477270 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:05:20.477865 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:05:20.478489 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:05:20.478507 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:05:20.479852 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:05:20.479856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:05:20.481236 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:05:20.481251 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:05:20.482085 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:05:20.482092 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:05:20.482354 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:05:20.483070 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:05:20.483094 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:05:20.485494 (+ 2400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:05:20.489139 (+ 3645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:05:20.489149 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:05:20.489165 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:05:20.489719 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:05:20.490186 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:05:20.490190 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:05:20.490192 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:05:20.490409 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:05:20.491046 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:05:20.491054 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:05:20.491429 (+ 375us) client_negotiation.cc:769] Sending connection context
0807 07:05:20.491688 (+ 259us) client_negotiation.cc:240] Negotiation successful
0807 07:05:20.492143 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":363,"threads_started":1}
I20260807 07:05:20.495505 12165 heartbeater.cc:347] Connected to a master server at 127.21.83.62:39001
I20260807 07:05:20.495842 12165 heartbeater.cc:464] Registering TS with master...
I20260807 07:05:20.496598 12165 heartbeater.cc:511] Master 127.21.83.62:39001 requested a full tablet report, sending...
I20260807 07:05:20.499226 11903 ts_manager.cc:194] Registered new tserver with Master: faefba3fc057402dae24f9e78dfdf048 (127.21.83.1:32771)
I20260807 07:05:20.501339 11903 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.1@KRBTEST.COM'} at 127.21.83.1:56865
WARNING: no policy specified for kudu/127.21.83.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab.
Entry for principal kudu/127.21.83.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab.
I20260807 07:05:20.545405 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.2:0
--local_ip_for_outbound_sockets=127.21.83.2
--webserver_interface=127.21.83.2
--webserver_port=0
--tserver_master_addrs=127.21.83.62:39001
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab
--principal=kudu/127.21.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:38161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:05:20.879920 12174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:05:20.880182 12174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:05:20.880313 12174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:05:20.887465 12174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:05:20.887573 12174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:05:20.887722 12174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.2
I20260807 07:05:20.897369 12174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:38161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab
--principal=kudu/127.21.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.83.2
--webserver_port=0
--tserver_master_addrs=127.21.83.62:39001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.2
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:05:20.898581 12174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:05:20.899870 12174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:05:20.912333 12182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:05:20.912246 12185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:05:20.912283 12183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:05:20.913357 12174 server_base.cc:1034] running on GCE node
I20260807 07:05:20.913875 12174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:05:20.914701 12174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:05:20.915891 12174 hybrid_clock.cc:648] HybridClock initialized: now 1786086320915847 us; error 38 us; skew 500 ppm
I20260807 07:05:20.916172 12174 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:05:20 dist-test-slave-ncp9 krb5kdc[11828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086320, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:05:20.919157 12174 init.cc:377] Logged in from keytab as kudu/127.21.83.2@KRBTEST.COM (short username kudu)
I20260807 07:05:20.920456 12174 webserver.cc:460] Webserver started at http://127.21.83.2:45977/ using document root <none> and password file <none>
I20260807 07:05:20.921123 12174 fs_manager.cc:359] Metadata directory not provided
I20260807 07:05:20.921242 12174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:05:20.921535 12174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:05:20.924280 12174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-1/data/instance:
uuid: "2c62ec1636f34095a1fa40b04f80da9e"
format_stamp: "Formatted at 2026-08-07 07:05:20 on dist-test-slave-ncp9"
server_key: "d50e3a0215607a0e4ffa9fdf98d2671b"
server_key_iv: "fd4e230fae2fb90a9775d32ffa2ccdb9"
server_key_version: "encryptionkey@0"
I20260807 07:05:20.925045 12174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-1/wal/instance:
uuid: "2c62ec1636f34095a1fa40b04f80da9e"
format_stamp: "Formatted at 2026-08-07 07:05:20 on dist-test-slave-ncp9"
server_key: "d50e3a0215607a0e4ffa9fdf98d2671b"
server_key_iv: "fd4e230fae2fb90a9775d32ffa2ccdb9"
server_key_version: "encryptionkey@0"
I20260807 07:05:20.929776 12174 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 07:05:20.932997 12194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:05:20.934363 12174 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 07:05:20.934527 12174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-1/wal
uuid: "2c62ec1636f34095a1fa40b04f80da9e"
format_stamp: "Formatted at 2026-08-07 07:05:20 on dist-test-slave-ncp9"
server_key: "d50e3a0215607a0e4ffa9fdf98d2671b"
server_key_iv: "fd4e230fae2fb90a9775d32ffa2ccdb9"
server_key_version: "encryptionkey@0"
I20260807 07:05:20.934722 12174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:05:20.951663 12174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:05:20.956287 12174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:05:20.956598 12174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:05:20.957722 12174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:05:20.959375 12174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:05:20.959458 12174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:05:20.959605 12174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:05:20.959661 12174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:05:20.991264 12174 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.2:46577
I20260807 07:05:20.991305 12350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.2:46577 every 8 connection(s)
I20260807 07:05:20.992515 12174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
I20260807 07:05:20.997193 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 12174
I20260807 07:05:20.997401 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-1/wal/instance
I20260807 07:05:20.997953 21836 external_mini_cluster.cc:1428] Setting key ff2410283f4a502465d0b5f5b2f84d31
Aug 07 07:05:21 dist-test-slave-ncp9 krb5kdc[11828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086320, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.2@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:05:21.015375 12355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:20.995424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:35771 (local address 127.21.83.62:39001)
0807 07:05:20.995892 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0807 07:05:20.995896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:05:20.996954 (+ 1058us) server_negotiation.cc:407] Connection header received
0807 07:05:20.999205 (+ 2251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:05:20.999210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:05:20.999293 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:05:20.999438 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:05:21.001414 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:21.002143 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:21.004294 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:21.004742 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:21.004839 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:05:21.009165 (+ 4326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:05:21.009170 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:05:21.009173 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:05:21.009210 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:05:21.011779 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:05:21.012745 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:05:21.012749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:05:21.012752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:05:21.012920 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:05:21.013444 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:05:21.013448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:05:21.013450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:05:21.013739 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:05:21.013856 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0807 07:05:21.014754 (+ 898us) server_negotiation.cc:299] Negotiation successful
0807 07:05:21.014977 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":252,"threads_started":1}
I20260807 07:05:21.016378 12356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:20.995837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.2:35771)
0807 07:05:20.996593 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:20.996641 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 07:05:20.998882 (+ 2241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:05:20.999673 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:05:20.999682 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:05:21.000341 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:05:21.001210 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:05:21.001238 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:05:21.002326 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:05:21.002332 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:05:21.004133 (+ 1801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:05:21.004149 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:05:21.004993 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:05:21.005001 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:05:21.005277 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:05:21.006056 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:05:21.006083 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:05:21.008880 (+ 2797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:05:21.011960 (+ 3080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:05:21.011969 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:05:21.011987 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:05:21.012599 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:05:21.013056 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:05:21.013060 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:05:21.013064 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:05:21.013326 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:05:21.013856 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:05:21.013863 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:05:21.014316 (+ 453us) client_negotiation.cc:769] Sending connection context
0807 07:05:21.014622 (+ 306us) client_negotiation.cc:240] Negotiation successful
0807 07:05:21.015203 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":300,"threads_started":1}
I20260807 07:05:21.017899 12351 heartbeater.cc:347] Connected to a master server at 127.21.83.62:39001
I20260807 07:05:21.018200 12351 heartbeater.cc:464] Registering TS with master...
I20260807 07:05:21.018966 12351 heartbeater.cc:511] Master 127.21.83.62:39001 requested a full tablet report, sending...
I20260807 07:05:21.020538 11903 ts_manager.cc:194] Registered new tserver with Master: 2c62ec1636f34095a1fa40b04f80da9e (127.21.83.2:46577)
I20260807 07:05:21.021427 11903 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.2@KRBTEST.COM'} at 127.21.83.2:35771
WARNING: no policy specified for kudu/127.21.83.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab.
Entry for principal kudu/127.21.83.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab.
I20260807 07:05:21.069707 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.3:0
--local_ip_for_outbound_sockets=127.21.83.3
--webserver_interface=127.21.83.3
--webserver_port=0
--tserver_master_addrs=127.21.83.62:39001
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab
--principal=kudu/127.21.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:38161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:05:21.407681 12359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:05:21.407979 12359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:05:21.408119 12359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:05:21.415359 12359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:05:21.415488 12359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:05:21.415645 12359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.3
I20260807 07:05:21.425500 12359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:38161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab
--principal=kudu/127.21.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.83.3
--webserver_port=0
--tserver_master_addrs=127.21.83.62:39001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.3
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:05:21.426820 12359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:05:21.428102 12359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:05:21.441159 12370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:05:21.441267 12366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:05:21.441331 12367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:05:21.442198 12359 server_base.cc:1034] running on GCE node
I20260807 07:05:21.442787 12359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:05:21.443949 12359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:05:21.445192 12359 hybrid_clock.cc:648] HybridClock initialized: now 1786086321445111 us; error 59 us; skew 500 ppm
I20260807 07:05:21.445556 12359 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:05:21 dist-test-slave-ncp9 krb5kdc[11828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086321, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:05:21.449659 12359 init.cc:377] Logged in from keytab as kudu/127.21.83.3@KRBTEST.COM (short username kudu)
I20260807 07:05:21.451543 12359 webserver.cc:460] Webserver started at http://127.21.83.3:41749/ using document root <none> and password file <none>
I20260807 07:05:21.452173 12359 fs_manager.cc:359] Metadata directory not provided
I20260807 07:05:21.452286 12359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:05:21.452567 12359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:05:21.455314 12359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-2/data/instance:
uuid: "0c43f4e5f9414fa6b99d9c0a7c0dd30c"
format_stamp: "Formatted at 2026-08-07 07:05:21 on dist-test-slave-ncp9"
server_key: "044ff0c34bf96ac04c77383f7f3a477a"
server_key_iv: "f7f629e11f8723ad1e524c108555ca30"
server_key_version: "encryptionkey@0"
I20260807 07:05:21.456007 12359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-2/wal/instance:
uuid: "0c43f4e5f9414fa6b99d9c0a7c0dd30c"
format_stamp: "Formatted at 2026-08-07 07:05:21 on dist-test-slave-ncp9"
server_key: "044ff0c34bf96ac04c77383f7f3a477a"
server_key_iv: "f7f629e11f8723ad1e524c108555ca30"
server_key_version: "encryptionkey@0"
I20260807 07:05:21.460669 12359 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260807 07:05:21.463794 12378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:05:21.464937 12359 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 07:05:21.465083 12359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-2/wal
uuid: "0c43f4e5f9414fa6b99d9c0a7c0dd30c"
format_stamp: "Formatted at 2026-08-07 07:05:21 on dist-test-slave-ncp9"
server_key: "044ff0c34bf96ac04c77383f7f3a477a"
server_key_iv: "f7f629e11f8723ad1e524c108555ca30"
server_key_version: "encryptionkey@0"
I20260807 07:05:21.465242 12359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:05:21.488926 12359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:05:21.492926 12359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:05:21.493201 12359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:05:21.494154 12359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:05:21.495486 12359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:05:21.495563 12359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:05:21.495682 12359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:05:21.495736 12359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:05:21.504987 12165 heartbeater.cc:503] Master 127.21.83.62:39001 was elected leader, sending a full tablet report...
I20260807 07:05:21.526163 12359 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.3:40235
I20260807 07:05:21.526211 12542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.3:40235 every 8 connection(s)
I20260807 07:05:21.527415 12359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
I20260807 07:05:21.530933 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 12359
I20260807 07:05:21.531118 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-2/wal/instance
I20260807 07:05:21.531615 21836 external_mini_cluster.cc:1428] Setting key 2e65dae961d340ea665d121555106d50
Aug 07 07:05:21 dist-test-slave-ncp9 krb5kdc[11828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086321, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.3@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:05:21.549882 12547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:21.529985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:43605 (local address 127.21.83.62:39001)
0807 07:05:21.530422 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0807 07:05:21.530427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:05:21.531575 (+ 1148us) server_negotiation.cc:407] Connection header received
0807 07:05:21.533800 (+ 2225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:05:21.533805 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:05:21.533894 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:05:21.533996 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:05:21.535893 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:21.536737 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:21.538460 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:21.538905 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:21.539080 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:05:21.543185 (+ 4105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:05:21.543190 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:05:21.543193 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:05:21.543233 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:05:21.546341 (+ 3108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:05:21.547308 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:05:21.547313 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:05:21.547316 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:05:21.547480 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:05:21.548101 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:05:21.548106 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:05:21.548109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:05:21.548411 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:05:21.548545 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0807 07:05:21.549377 (+ 832us) server_negotiation.cc:299] Negotiation successful
0807 07:05:21.549566 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":206,"threads_started":1}
I20260807 07:05:21.550879 12548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:21.530559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.3:43605)
0807 07:05:21.531391 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:21.531439 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 07:05:21.533324 (+ 1885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:05:21.534332 (+ 1008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:05:21.534345 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:05:21.535062 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:05:21.535695 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:05:21.535715 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:05:21.536962 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:05:21.536969 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:05:21.538276 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:05:21.538293 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:05:21.539235 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:05:21.539244 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:05:21.539486 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:05:21.540175 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:05:21.540199 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:05:21.542920 (+ 2721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:05:21.546531 (+ 3611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:05:21.546540 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:05:21.546556 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:05:21.547173 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:05:21.547640 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:05:21.547645 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:05:21.547649 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:05:21.547951 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:05:21.548565 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:05:21.548573 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:05:21.549024 (+ 451us) client_negotiation.cc:769] Sending connection context
0807 07:05:21.549296 (+ 272us) client_negotiation.cc:240] Negotiation successful
0807 07:05:21.549737 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":337,"threads_started":1}
I20260807 07:05:21.552407 12545 heartbeater.cc:347] Connected to a master server at 127.21.83.62:39001
I20260807 07:05:21.552732 12545 heartbeater.cc:464] Registering TS with master...
I20260807 07:05:21.553509 12545 heartbeater.cc:511] Master 127.21.83.62:39001 requested a full tablet report, sending...
I20260807 07:05:21.555047 11903 ts_manager.cc:194] Registered new tserver with Master: 0c43f4e5f9414fa6b99d9c0a7c0dd30c (127.21.83.3:40235)
I20260807 07:05:21.555867 11903 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.3@KRBTEST.COM'} at 127.21.83.3:43605
I20260807 07:05:21.560658 21836 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 07:05:21.577942 12547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:21.565290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48628 (local address 127.21.83.62:39001)
0807 07:05:21.565533 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0807 07:05:21.565540 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:05:21.565917 (+ 377us) server_negotiation.cc:407] Connection header received
0807 07:05:21.566054 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:05:21.566058 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:05:21.566129 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:05:21.566206 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:05:21.567753 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:21.568589 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:21.570030 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:21.570445 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:21.570541 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:05:21.572840 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:05:21.572845 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:05:21.572849 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:05:21.572881 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:05:21.575493 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:05:21.576141 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:05:21.576144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:05:21.576147 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:05:21.576269 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:05:21.576623 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:05:21.576627 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:05:21.576629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:05:21.577032 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:05:21.577223 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0807 07:05:21.577500 (+ 277us) server_negotiation.cc:299] Negotiation successful
0807 07:05:21.577669 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 07:05:21.581858 11903 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48628:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 07:05:21.588626 11903 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 07:05:21.624044 12562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:21.601247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.2:46577 (local address 127.0.0.1:57064)
0807 07:05:21.608041 (+ 6794us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:21.608091 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0807 07:05:21.608302 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:05:21.615861 (+ 7559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:05:21.615869 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:05:21.615901 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:05:21.616321 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:05:21.616334 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:05:21.618282 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:05:21.618288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:05:21.621115 (+ 2827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:05:21.621135 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:05:21.622970 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:05:21.622989 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:05:21.623170 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:05:21.623224 (+ 54us) client_negotiation.cc:769] Sending connection context
0807 07:05:21.623330 (+ 106us) client_negotiation.cc:240] Negotiation successful
0807 07:05:21.623554 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6485,"mutex_wait_us":131,"thread_start_us":6301,"threads_started":1}
I20260807 07:05:21.624513 12569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:21.613989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:32771 (local address 127.0.0.1:43814)
0807 07:05:21.615516 (+ 1527us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:21.615534 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 07:05:21.615646 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:05:21.617047 (+ 1401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:05:21.617052 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:05:21.617077 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:05:21.617425 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:05:21.617437 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:05:21.619272 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:05:21.619278 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:05:21.621499 (+ 2221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:05:21.621510 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:05:21.623412 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:05:21.623417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:05:21.623602 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:05:21.623633 (+ 31us) client_negotiation.cc:769] Sending connection context
0807 07:05:21.623717 (+ 84us) client_negotiation.cc:240] Negotiation successful
0807 07:05:21.623842 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1408,"mutex_wait_us":103,"thread_start_us":1262,"threads_started":1}
I20260807 07:05:21.624936 12567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:21.609005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43814 (local address 127.21.83.1:32771)
0807 07:05:21.616238 (+ 7233us) server_negotiation.cc:206] Beginning negotiation
0807 07:05:21.616246 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 07:05:21.616276 (+ 30us) server_negotiation.cc:407] Connection header received
0807 07:05:21.616367 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:05:21.616374 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:05:21.616598 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:05:21.616777 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:05:21.617593 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:21.619097 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:21.621650 (+ 2553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:21.622673 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:21.622805 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:05:21.623053 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0807 07:05:21.624379 (+ 1326us) server_negotiation.cc:299] Negotiation successful
0807 07:05:21.624567 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7048,"thread_start_us":6862,"threads_started":1}
I20260807 07:05:21.625021 12564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:21.601922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.3:40235 (local address 127.0.0.1:38152)
0807 07:05:21.614735 (+ 12813us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:21.614757 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 07:05:21.614915 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:05:21.618178 (+ 3263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:05:21.618183 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:05:21.618209 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:05:21.618684 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:05:21.618694 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:05:21.620413 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:05:21.620417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:05:21.622414 (+ 1997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:05:21.622430 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:05:21.624163 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:05:21.624169 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:05:21.624337 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:05:21.624369 (+ 32us) client_negotiation.cc:769] Sending connection context
0807 07:05:21.624460 (+ 91us) client_negotiation.cc:240] Negotiation successful
0807 07:05:21.624588 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12671,"mutex_wait_us":83,"thread_start_us":12506,"threads_started":1}
I20260807 07:05:21.625119 12563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:21.601460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57064 (local address 127.21.83.2:46577)
0807 07:05:21.615124 (+ 13664us) server_negotiation.cc:206] Beginning negotiation
0807 07:05:21.615135 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 07:05:21.615173 (+ 38us) server_negotiation.cc:407] Connection header received
0807 07:05:21.615293 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:05:21.615302 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:05:21.615610 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:05:21.616544 (+ 934us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:05:21.616768 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:21.618132 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:21.621311 (+ 3179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:21.622754 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:21.624318 (+ 1564us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:05:21.624504 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0807 07:05:21.624586 (+ 82us) server_negotiation.cc:299] Negotiation successful
0807 07:05:21.624751 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13511,"thread_start_us":13365,"threads_started":1}
I20260807 07:05:21.625728 12565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:21.602123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38152 (local address 127.21.83.3:40235)
0807 07:05:21.617622 (+ 15499us) server_negotiation.cc:206] Beginning negotiation
0807 07:05:21.617630 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 07:05:21.617664 (+ 34us) server_negotiation.cc:407] Connection header received
0807 07:05:21.617758 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:05:21.617765 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:05:21.617976 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:05:21.618171 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:05:21.618860 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:21.620261 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:21.622603 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:21.623881 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:21.624011 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:05:21.624240 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0807 07:05:21.625214 (+ 974us) server_negotiation.cc:299] Negotiation successful
0807 07:05:21.625390 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15332,"thread_start_us":15153,"threads_started":1}
I20260807 07:05:21.628832 12080 tablet_service.cc:1467] Processing CreateTablet for tablet 86ca1107ae564a6084fdb81af977419d (DEFAULT_TABLE table=test-table [id=e30d1a6d5c4f484a8547bb824a8ae907]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:05:21.628876 12260 tablet_service.cc:1467] Processing CreateTablet for tablet 86ca1107ae564a6084fdb81af977419d (DEFAULT_TABLE table=test-table [id=e30d1a6d5c4f484a8547bb824a8ae907]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:05:21.630223 12080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86ca1107ae564a6084fdb81af977419d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:05:21.630343 12260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86ca1107ae564a6084fdb81af977419d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:05:21.630846 12444 tablet_service.cc:1467] Processing CreateTablet for tablet 86ca1107ae564a6084fdb81af977419d (DEFAULT_TABLE table=test-table [id=e30d1a6d5c4f484a8547bb824a8ae907]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:05:21.632360 12444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86ca1107ae564a6084fdb81af977419d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:05:21.642180 12570 tablet_bootstrap.cc:492] T 86ca1107ae564a6084fdb81af977419d P faefba3fc057402dae24f9e78dfdf048: Bootstrap starting.
I20260807 07:05:21.644367 12571 tablet_bootstrap.cc:492] T 86ca1107ae564a6084fdb81af977419d P 2c62ec1636f34095a1fa40b04f80da9e: Bootstrap starting.
I20260807 07:05:21.646369 12570 tablet_bootstrap.cc:654] T 86ca1107ae564a6084fdb81af977419d P faefba3fc057402dae24f9e78dfdf048: Neither blocks nor log segments found. Creating new log.
I20260807 07:05:21.647198 12573 tablet_bootstrap.cc:492] T 86ca1107ae564a6084fdb81af977419d P 0c43f4e5f9414fa6b99d9c0a7c0dd30c: Bootstrap starting.
I20260807 07:05:21.647738 12571 tablet_bootstrap.cc:654] T 86ca1107ae564a6084fdb81af977419d P 2c62ec1636f34095a1fa40b04f80da9e: Neither blocks nor log segments found. Creating new log.
I20260807 07:05:21.647773 12570 log.cc:824] T 86ca1107ae564a6084fdb81af977419d P faefba3fc057402dae24f9e78dfdf048: Log is configured to *not* fsync() on all Append() calls
I20260807 07:05:21.648821 12571 log.cc:824] T 86ca1107ae564a6084fdb81af977419d P 2c62ec1636f34095a1fa40b04f80da9e: Log is configured to *not* fsync() on all Append() calls
I20260807 07:05:21.650115 12573 tablet_bootstrap.cc:654] T 86ca1107ae564a6084fdb81af977419d P 0c43f4e5f9414fa6b99d9c0a7c0dd30c: Neither blocks nor log segments found. Creating new log.
I20260807 07:05:21.650729 12570 tablet_bootstrap.cc:492] T 86ca1107ae564a6084fdb81af977419d P faefba3fc057402dae24f9e78dfdf048: No bootstrap required, opened a new log
I20260807 07:05:21.651014 12570 ts_tablet_manager.cc:1397] T 86ca1107ae564a6084fdb81af977419d P faefba3fc057402dae24f9e78dfdf048: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260807 07:05:21.651012 12573 log.cc:824] T 86ca1107ae564a6084fdb81af977419d P 0c43f4e5f9414fa6b99d9c0a7c0dd30c: Log is configured to *not* fsync() on all Append() calls
I20260807 07:05:21.651623 12571 tablet_bootstrap.cc:492] T 86ca1107ae564a6084fdb81af977419d P 2c62ec1636f34095a1fa40b04f80da9e: No bootstrap required, opened a new log
I20260807 07:05:21.651870 12571 ts_tablet_manager.cc:1397] T 86ca1107ae564a6084fdb81af977419d P 2c62ec1636f34095a1fa40b04f80da9e: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260807 07:05:21.653367 12573 tablet_bootstrap.cc:492] T 86ca1107ae564a6084fdb81af977419d P 0c43f4e5f9414fa6b99d9c0a7c0dd30c: No bootstrap required, opened a new log
I20260807 07:05:21.653589 12573 ts_tablet_manager.cc:1397] T 86ca1107ae564a6084fdb81af977419d P 0c43f4e5f9414fa6b99d9c0a7c0dd30c: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260807 07:05:21.660386 12573 raft_consensus.cc:348] T 86ca1107ae564a6084fdb81af977419d P 0c43f4e5f9414fa6b99d9c0a7c0dd30c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c62ec1636f34095a1fa40b04f80da9e" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 46577 } } peers { permanent_uuid: "faefba3fc057402dae24f9e78dfdf048" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 32771 } } peers { permanent_uuid: "0c43f4e5f9414fa6b99d9c0a7c0dd30c" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40235 } }
I20260807 07:05:21.660730 12573 raft_consensus.cc:374] T 86ca1107ae564a6084fdb81af977419d P 0c43f4e5f9414fa6b99d9c0a7c0dd30c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:05:21.660825 12573 raft_consensus.cc:729] T 86ca1107ae564a6084fdb81af977419d P 0c43f4e5f9414fa6b99d9c0a7c0dd30c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c43f4e5f9414fa6b99d9c0a7c0dd30c, State: Initialized, Role: FOLLOWER
I20260807 07:05:21.661144 12570 raft_consensus.cc:348] T 86ca1107ae564a6084fdb81af977419d P faefba3fc057402dae24f9e78dfdf048 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c62ec1636f34095a1fa40b04f80da9e" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 46577 } } peers { permanent_uuid: "faefba3fc057402dae24f9e78dfdf048" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 32771 } } peers { permanent_uuid: "0c43f4e5f9414fa6b99d9c0a7c0dd30c" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40235 } }
I20260807 07:05:21.661526 12570 raft_consensus.cc:374] T 86ca1107ae564a6084fdb81af977419d P faefba3fc057402dae24f9e78dfdf048 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:05:21.661386 12573 consensus_queue.cc:260] T 86ca1107ae564a6084fdb81af977419d P 0c43f4e5f9414fa6b99d9c0a7c0dd30c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c62ec1636f34095a1fa40b04f80da9e" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 46577 } } peers { permanent_uuid: "faefba3fc057402dae24f9e78dfdf048" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 32771 } } peers { permanent_uuid: "0c43f4e5f9414fa6b99d9c0a7c0dd30c" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40235 } }
I20260807 07:05:21.661634 12570 raft_consensus.cc:729] T 86ca1107ae564a6084fdb81af977419d P faefba3fc057402dae24f9e78dfdf048 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faefba3fc057402dae24f9e78dfdf048, State: Initialized, Role: FOLLOWER
I20260807 07:05:21.661753 12571 raft_consensus.cc:348] T 86ca1107ae564a6084fdb81af977419d P 2c62ec1636f34095a1fa40b04f80da9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c62ec1636f34095a1fa40b04f80da9e" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 46577 } } peers { permanent_uuid: "faefba3fc057402dae24f9e78dfdf048" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 32771 } } peers { permanent_uuid: "0c43f4e5f9414fa6b99d9c0a7c0dd30c" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40235 } }
I20260807 07:05:21.662205 12571 raft_consensus.cc:374] T 86ca1107ae564a6084fdb81af977419d P 2c62ec1636f34095a1fa40b04f80da9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:05:21.662312 12571 raft_consensus.cc:729] T 86ca1107ae564a6084fdb81af977419d P 2c62ec1636f34095a1fa40b04f80da9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c62ec1636f34095a1fa40b04f80da9e, State: Initialized, Role: FOLLOWER
I20260807 07:05:21.662210 12570 consensus_queue.cc:260] T 86ca1107ae564a6084fdb81af977419d P faefba3fc057402dae24f9e78dfdf048 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c62ec1636f34095a1fa40b04f80da9e" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 46577 } } peers { permanent_uuid: "faefba3fc057402dae24f9e78dfdf048" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 32771 } } peers { permanent_uuid: "0c43f4e5f9414fa6b99d9c0a7c0dd30c" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40235 } }
I20260807 07:05:21.662887 12545 heartbeater.cc:503] Master 127.21.83.62:39001 was elected leader, sending a full tablet report...
I20260807 07:05:21.663008 12571 consensus_queue.cc:260] T 86ca1107ae564a6084fdb81af977419d P 2c62ec1636f34095a1fa40b04f80da9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c62ec1636f34095a1fa40b04f80da9e" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 46577 } } peers { permanent_uuid: "faefba3fc057402dae24f9e78dfdf048" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 32771 } } peers { permanent_uuid: "0c43f4e5f9414fa6b99d9c0a7c0dd30c" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40235 } }
I20260807 07:05:21.663947 12570 ts_tablet_manager.cc:1428] T 86ca1107ae564a6084fdb81af977419d P faefba3fc057402dae24f9e78dfdf048: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260807 07:05:21.664779 12351 heartbeater.cc:503] Master 127.21.83.62:39001 was elected leader, sending a full tablet report...
I20260807 07:05:21.665395 12571 ts_tablet_manager.cc:1428] T 86ca1107ae564a6084fdb81af977419d P 2c62ec1636f34095a1fa40b04f80da9e: Time spent starting tablet: real 0.013s user 0.014s sys 0.001s
I20260807 07:05:21.666985 12573 ts_tablet_manager.cc:1428] T 86ca1107ae564a6084fdb81af977419d P 0c43f4e5f9414fa6b99d9c0a7c0dd30c: Time spent starting tablet: real 0.013s user 0.006s sys 0.004s
W20260807 07:05:21.722805 12166 tablet.cc:2369] T 86ca1107ae564a6084fdb81af977419d P faefba3fc057402dae24f9e78dfdf048: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 07:05:21.744580 12352 tablet.cc:2369] T 86ca1107ae564a6084fdb81af977419d P 2c62ec1636f34095a1fa40b04f80da9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 07:05:21.779043 12546 tablet.cc:2369] T 86ca1107ae564a6084fdb81af977419d P 0c43f4e5f9414fa6b99d9c0a7c0dd30c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 07:05:21.785491 12584 raft_consensus.cc:482] T 86ca1107ae564a6084fdb81af977419d P 2c62ec1636f34095a1fa40b04f80da9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:05:21.785786 12584 raft_consensus.cc:504] T 86ca1107ae564a6084fdb81af977419d P 2c62ec1636f34095a1fa40b04f80da9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c62ec1636f34095a1fa40b04f80da9e" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 46577 } } peers { permanent_uuid: "faefba3fc057402dae24f9e78dfdf048" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 32771 } } peers { permanent_uuid: "0c43f4e5f9414fa6b99d9c0a7c0dd30c" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40235 } }
I20260807 07:05:21.787585 12584 leader_election.cc:288] T 86ca1107ae564a6084fdb81af977419d P 2c62ec1636f34095a1fa40b04f80da9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers faefba3fc057402dae24f9e78dfdf048 (127.21.83.1:32771), 0c43f4e5f9414fa6b99d9c0a7c0dd30c (127.21.83.3:40235)
I20260807 07:05:21.793907 12567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:21.787997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:55827 (local address 127.21.83.1:32771)
0807 07:05:21.788216 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0807 07:05:21.788223 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:05:21.788697 (+ 474us) server_negotiation.cc:407] Connection header received
0807 07:05:21.788914 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:05:21.788921 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:05:21.788997 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:05:21.789072 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:05:21.789621 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:21.790336 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:21.792120 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:21.792926 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:21.793079 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:05:21.793185 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0807 07:05:21.793521 (+ 336us) server_negotiation.cc:299] Negotiation successful
0807 07:05:21.793660 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 07:05:21.794041 12585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:21.787863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:32771 (local address 127.21.83.2:55827)
0807 07:05:21.788584 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:21.788600 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 07:05:21.788755 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:05:21.789103 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:05:21.789108 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:05:21.789133 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:05:21.789482 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:05:21.789493 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:05:21.790468 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:05:21.790473 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:05:21.791967 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:05:21.791980 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:05:21.793167 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:05:21.793173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:05:21.793339 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:05:21.793386 (+ 47us) client_negotiation.cc:769] Sending connection context
0807 07:05:21.793477 (+ 91us) client_negotiation.cc:240] Negotiation successful
0807 07:05:21.793601 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":6,"thread_start_us":358,"threads_started":1}
I20260807 07:05:21.794059 12565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:21.788096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:35031 (local address 127.21.83.3:40235)
0807 07:05:21.788291 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0807 07:05:21.788298 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:05:21.788918 (+ 620us) server_negotiation.cc:407] Connection header received
0807 07:05:21.789093 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:05:21.789098 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:05:21.789160 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:05:21.789233 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:05:21.789832 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:21.790531 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:21.792238 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:21.793072 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:21.793195 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:05:21.793302 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0807 07:05:21.793681 (+ 379us) server_negotiation.cc:299] Negotiation successful
0807 07:05:21.793810 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 07:05:21.794315 12586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:21.788017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.3:40235 (local address 127.21.83.2:35031)
0807 07:05:21.788793 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:21.788807 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 07:05:21.788955 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:05:21.789277 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:05:21.789281 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:05:21.789309 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:05:21.789688 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:05:21.789697 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:05:21.790672 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:05:21.790676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:05:21.792087 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:05:21.792100 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:05:21.793336 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:05:21.793341 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:05:21.793521 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:05:21.793549 (+ 28us) client_negotiation.cc:769] Sending connection context
0807 07:05:21.793630 (+ 81us) client_negotiation.cc:240] Negotiation successful
0807 07:05:21.793751 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"mutex_wait_us":33,"thread_start_us":453,"threads_started":1}
I20260807 07:05:21.795223 12107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86ca1107ae564a6084fdb81af977419d" candidate_uuid: "2c62ec1636f34095a1fa40b04f80da9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faefba3fc057402dae24f9e78dfdf048" is_pre_election: true
I20260807 07:05:21.795301 12468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86ca1107ae564a6084fdb81af977419d" candidate_uuid: "2c62ec1636f34095a1fa40b04f80da9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c43f4e5f9414fa6b99d9c0a7c0dd30c" is_pre_election: true
I20260807 07:05:21.795830 12468 raft_consensus.cc:2454] T 86ca1107ae564a6084fdb81af977419d P 0c43f4e5f9414fa6b99d9c0a7c0dd30c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c62ec1636f34095a1fa40b04f80da9e in term 0.
I20260807 07:05:21.795830 12107 raft_consensus.cc:2454] T 86ca1107ae564a6084fdb81af977419d P faefba3fc057402dae24f9e78dfdf048 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c62ec1636f34095a1fa40b04f80da9e in term 0.
I20260807 07:05:21.796591 12198 leader_election.cc:302] T 86ca1107ae564a6084fdb81af977419d P 2c62ec1636f34095a1fa40b04f80da9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c43f4e5f9414fa6b99d9c0a7c0dd30c, 2c62ec1636f34095a1fa40b04f80da9e; no voters:
I20260807 07:05:21.797042 12584 raft_consensus.cc:2781] T 86ca1107ae564a6084fdb81af977419d P 2c62ec1636f34095a1fa40b04f80da9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 07:05:21.797185 12584 raft_consensus.cc:482] T 86ca1107ae564a6084fdb81af977419d P 2c62ec1636f34095a1fa40b04f80da9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:05:21.797278 12584 raft_consensus.cc:3037] T 86ca1107ae564a6084fdb81af977419d P 2c62ec1636f34095a1fa40b04f80da9e [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:05:21.799436 12584 raft_consensus.cc:504] T 86ca1107ae564a6084fdb81af977419d P 2c62ec1636f34095a1fa40b04f80da9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c62ec1636f34095a1fa40b04f80da9e" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 46577 } } peers { permanent_uuid: "faefba3fc057402dae24f9e78dfdf048" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 32771 } } peers { permanent_uuid: "0c43f4e5f9414fa6b99d9c0a7c0dd30c" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40235 } }
I20260807 07:05:21.799994 12584 leader_election.cc:288] T 86ca1107ae564a6084fdb81af977419d P 2c62ec1636f34095a1fa40b04f80da9e [CANDIDATE]: Term 1 election: Requested vote from peers faefba3fc057402dae24f9e78dfdf048 (127.21.83.1:32771), 0c43f4e5f9414fa6b99d9c0a7c0dd30c (127.21.83.3:40235)
I20260807 07:05:21.800504 12107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86ca1107ae564a6084fdb81af977419d" candidate_uuid: "2c62ec1636f34095a1fa40b04f80da9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faefba3fc057402dae24f9e78dfdf048"
I20260807 07:05:21.800522 12468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86ca1107ae564a6084fdb81af977419d" candidate_uuid: "2c62ec1636f34095a1fa40b04f80da9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c43f4e5f9414fa6b99d9c0a7c0dd30c"
I20260807 07:05:21.800693 12107 raft_consensus.cc:3037] T 86ca1107ae564a6084fdb81af977419d P faefba3fc057402dae24f9e78dfdf048 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:05:21.800698 12468 raft_consensus.cc:3037] T 86ca1107ae564a6084fdb81af977419d P 0c43f4e5f9414fa6b99d9c0a7c0dd30c [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:05:21.802915 12468 raft_consensus.cc:2454] T 86ca1107ae564a6084fdb81af977419d P 0c43f4e5f9414fa6b99d9c0a7c0dd30c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c62ec1636f34095a1fa40b04f80da9e in term 1.
I20260807 07:05:21.802917 12107 raft_consensus.cc:2454] T 86ca1107ae564a6084fdb81af977419d P faefba3fc057402dae24f9e78dfdf048 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c62ec1636f34095a1fa40b04f80da9e in term 1.
I20260807 07:05:21.803388 12199 leader_election.cc:302] T 86ca1107ae564a6084fdb81af977419d P 2c62ec1636f34095a1fa40b04f80da9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0c43f4e5f9414fa6b99d9c0a7c0dd30c, 2c62ec1636f34095a1fa40b04f80da9e, faefba3fc057402dae24f9e78dfdf048; no voters:
I20260807 07:05:21.803740 12584 raft_consensus.cc:2781] T 86ca1107ae564a6084fdb81af977419d P 2c62ec1636f34095a1fa40b04f80da9e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:05:21.804196 12584 raft_consensus.cc:686] T 86ca1107ae564a6084fdb81af977419d P 2c62ec1636f34095a1fa40b04f80da9e [term 1 LEADER]: Becoming Leader. State: Replica: 2c62ec1636f34095a1fa40b04f80da9e, State: Running, Role: LEADER
I20260807 07:05:21.804667 12584 consensus_queue.cc:237] T 86ca1107ae564a6084fdb81af977419d P 2c62ec1636f34095a1fa40b04f80da9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c62ec1636f34095a1fa40b04f80da9e" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 46577 } } peers { permanent_uuid: "faefba3fc057402dae24f9e78dfdf048" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 32771 } } peers { permanent_uuid: "0c43f4e5f9414fa6b99d9c0a7c0dd30c" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40235 } }
I20260807 07:05:21.809332 11902 catalog_manager.cc:5495] T 86ca1107ae564a6084fdb81af977419d P 2c62ec1636f34095a1fa40b04f80da9e reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c62ec1636f34095a1fa40b04f80da9e (127.21.83.2). New cstate: current_term: 1 leader_uuid: "2c62ec1636f34095a1fa40b04f80da9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c62ec1636f34095a1fa40b04f80da9e" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 46577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "faefba3fc057402dae24f9e78dfdf048" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 32771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c43f4e5f9414fa6b99d9c0a7c0dd30c" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40235 } health_report { overall_health: UNKNOWN } } }
I20260807 07:05:21.865725 12563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:21.860261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57080 (local address 127.21.83.2:46577)
0807 07:05:21.860455 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0807 07:05:21.860462 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:05:21.860512 (+ 50us) server_negotiation.cc:407] Connection header received
0807 07:05:21.860781 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:05:21.860786 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:05:21.860864 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:05:21.861011 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:05:21.861392 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:21.862128 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:21.863615 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:21.864022 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:21.864128 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:05:21.864611 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:05:21.865111 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:05:21.865258 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0807 07:05:21.865353 (+ 95us) server_negotiation.cc:299] Negotiation successful
0807 07:05:21.865434 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 07:05:21.873334 12107 raft_consensus.cc:1261] T 86ca1107ae564a6084fdb81af977419d P faefba3fc057402dae24f9e78dfdf048 [term 1 FOLLOWER]: Refusing update from remote peer 2c62ec1636f34095a1fa40b04f80da9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:05:21.873350 12468 raft_consensus.cc:1261] T 86ca1107ae564a6084fdb81af977419d P 0c43f4e5f9414fa6b99d9c0a7c0dd30c [term 1 FOLLOWER]: Refusing update from remote peer 2c62ec1636f34095a1fa40b04f80da9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:05:21.874331 12589 consensus_queue.cc:1035] T 86ca1107ae564a6084fdb81af977419d P 2c62ec1636f34095a1fa40b04f80da9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c43f4e5f9414fa6b99d9c0a7c0dd30c" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:05:21.874701 12584 consensus_queue.cc:1035] T 86ca1107ae564a6084fdb81af977419d P 2c62ec1636f34095a1fa40b04f80da9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "faefba3fc057402dae24f9e78dfdf048" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 32771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:05:21.884370 12593 mvcc.cc:205] Tried to move back new op lower bound from 7315809574383505408 to 7315809574115635200. Current Snapshot: MvccSnapshot[applied={T|T < 7315809574383505408}]
I20260807 07:05:21.897887 11902 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:05:21.898190 11902 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:05:21.901688 11902 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5: Sending DeleteTablet for 3 replicas of tablet 86ca1107ae564a6084fdb81af977419d
I20260807 07:05:21.902814 12444 tablet_service.cc:1514] Processing DeleteTablet for tablet 86ca1107ae564a6084fdb81af977419d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:05:21 UTC) from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:38152
I20260807 07:05:21.902829 12260 tablet_service.cc:1514] Processing DeleteTablet for tablet 86ca1107ae564a6084fdb81af977419d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:05:21 UTC) from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:57064
I20260807 07:05:21.902802 12080 tablet_service.cc:1514] Processing DeleteTablet for tablet 86ca1107ae564a6084fdb81af977419d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:05:21 UTC) from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:43814
I20260807 07:05:21.903200 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 11993
I20260807 07:05:21.923444 12609 tablet_replica.cc:330] T 86ca1107ae564a6084fdb81af977419d P 0c43f4e5f9414fa6b99d9c0a7c0dd30c: stopping tablet replica
I20260807 07:05:21.923913 12609 raft_consensus.cc:2229] T 86ca1107ae564a6084fdb81af977419d P 0c43f4e5f9414fa6b99d9c0a7c0dd30c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:05:21.924443 12609 raft_consensus.cc:2258] T 86ca1107ae564a6084fdb81af977419d P 0c43f4e5f9414fa6b99d9c0a7c0dd30c [term 1 FOLLOWER]: Raft consensus is shut down!
Aug 07 07:05:21 dist-test-slave-ncp9 krb5kdc[11828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.1@KRBTEST.COM
I20260807 07:05:21.933444 12608 tablet_replica.cc:330] T 86ca1107ae564a6084fdb81af977419d P 2c62ec1636f34095a1fa40b04f80da9e: stopping tablet replica
I20260807 07:05:21.933792 12608 raft_consensus.cc:2229] T 86ca1107ae564a6084fdb81af977419d P 2c62ec1636f34095a1fa40b04f80da9e [term 1 LEADER]: Raft consensus shutting down.
I20260807 07:05:21.934365 12608 raft_consensus.cc:2258] T 86ca1107ae564a6084fdb81af977419d P 2c62ec1636f34095a1fa40b04f80da9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:05:21.935154 12610 tablet_replica.cc:330] T 86ca1107ae564a6084fdb81af977419d P faefba3fc057402dae24f9e78dfdf048: stopping tablet replica
I20260807 07:05:21.935534 12610 raft_consensus.cc:2229] T 86ca1107ae564a6084fdb81af977419d P faefba3fc057402dae24f9e78dfdf048 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:05:21.935634 12609 ts_tablet_manager.cc:1905] T 86ca1107ae564a6084fdb81af977419d P 0c43f4e5f9414fa6b99d9c0a7c0dd30c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:05:21.936022 12610 raft_consensus.cc:2258] T 86ca1107ae564a6084fdb81af977419d P faefba3fc057402dae24f9e78dfdf048 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:05:21.937968 12567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:21.904519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43818 (local address 127.21.83.1:32771)
0807 07:05:21.904708 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0807 07:05:21.904714 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:05:21.915740 (+ 11026us) server_negotiation.cc:407] Connection header received
0807 07:05:21.916188 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:05:21.916195 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:05:21.916281 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:05:21.916354 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:05:21.918647 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:21.919907 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:21.922065 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:21.922805 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:21.922904 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:05:21.929243 (+ 6339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:05:21.929259 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:05:21.929279 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:05:21.929332 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:05:21.934039 (+ 4707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:05:21.935026 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:05:21.935035 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:05:21.935039 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:05:21.935296 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:05:21.935918 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:05:21.935923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:05:21.935926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:05:21.936455 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:05:21.936736 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0807 07:05:21.937385 (+ 649us) server_negotiation.cc:299] Negotiation successful
0807 07:05:21.937600 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 07:05:21.939527 12610 ts_tablet_manager.cc:1905] T 86ca1107ae564a6084fdb81af977419d P faefba3fc057402dae24f9e78dfdf048: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:05:21.939670 12160 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:05:21.940320 12608 ts_tablet_manager.cc:1905] T 86ca1107ae564a6084fdb81af977419d P 2c62ec1636f34095a1fa40b04f80da9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:05:21.943094 12609 ts_tablet_manager.cc:1918] T 86ca1107ae564a6084fdb81af977419d P 0c43f4e5f9414fa6b99d9c0a7c0dd30c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:05:21.943259 12609 log.cc:1196] T 86ca1107ae564a6084fdb81af977419d P 0c43f4e5f9414fa6b99d9c0a7c0dd30c: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-2/wal/wals/86ca1107ae564a6084fdb81af977419d
I20260807 07:05:21.943796 12609 ts_tablet_manager.cc:1939] T 86ca1107ae564a6084fdb81af977419d P 0c43f4e5f9414fa6b99d9c0a7c0dd30c: Deleting consensus metadata
I20260807 07:05:21.945678 11877 catalog_manager.cc:4841] TS 0c43f4e5f9414fa6b99d9c0a7c0dd30c (127.21.83.3:40235): tablet 86ca1107ae564a6084fdb81af977419d (table test-table [id=e30d1a6d5c4f484a8547bb824a8ae907]) successfully deleted
I20260807 07:05:21.948339 12608 ts_tablet_manager.cc:1918] T 86ca1107ae564a6084fdb81af977419d P 2c62ec1636f34095a1fa40b04f80da9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:05:21.948529 12608 log.cc:1196] T 86ca1107ae564a6084fdb81af977419d P 2c62ec1636f34095a1fa40b04f80da9e: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-1/wal/wals/86ca1107ae564a6084fdb81af977419d
I20260807 07:05:21.949330 12608 ts_tablet_manager.cc:1939] T 86ca1107ae564a6084fdb81af977419d P 2c62ec1636f34095a1fa40b04f80da9e: Deleting consensus metadata
I20260807 07:05:21.950999 11878 catalog_manager.cc:4841] TS 2c62ec1636f34095a1fa40b04f80da9e (127.21.83.2:46577): tablet 86ca1107ae564a6084fdb81af977419d (table test-table [id=e30d1a6d5c4f484a8547bb824a8ae907]) successfully deleted
I20260807 07:05:22.098831 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 11993
I20260807 07:05:22.123252 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 12174
W20260807 07:05:22.123270 11878 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.83.1:32771 (error 108)
I20260807 07:05:22.125125 12562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:22.124550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:32771 (local address 127.0.0.1:43828)
0807 07:05:22.124726 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:22.124844 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.1:32771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260807 07:05:22.125408 11878 catalog_manager.cc:4584] TS faefba3fc057402dae24f9e78dfdf048 (127.21.83.1:32771): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 86ca1107ae564a6084fdb81af977419d: Network error: Client connection negotiation failed: client connection to 127.21.83.1:32771: connect: Connection refused (error 111)
Aug 07 07:05:22 dist-test-slave-ncp9 krb5kdc[11828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.2@KRBTEST.COM
I20260807 07:05:22.142789 12563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:22.124540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57096 (local address 127.21.83.2:46577)
0807 07:05:22.124860 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0807 07:05:22.124865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:05:22.124886 (+ 21us) server_negotiation.cc:407] Connection header received
0807 07:05:22.124988 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:05:22.124993 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:05:22.125074 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:05:22.125180 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:05:22.127100 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:22.127952 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:22.129897 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:22.130355 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:22.130451 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:05:22.136258 (+ 5807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:05:22.136278 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:05:22.136295 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:05:22.136344 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:05:22.139676 (+ 3332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:05:22.140662 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:05:22.140669 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:05:22.140672 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:05:22.140813 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:05:22.141218 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:05:22.141221 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:05:22.141223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:05:22.141614 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:05:22.141889 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0807 07:05:22.142289 (+ 400us) server_negotiation.cc:299] Negotiation successful
0807 07:05:22.142470 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195}
I20260807 07:05:22.143635 12346 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:05:22.179140 12562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:22.178624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:32771 (local address 127.0.0.1:43832)
0807 07:05:22.178859 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:22.178956 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.1:32771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260807 07:05:22.180187 12562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:22.179847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:32771 (local address 127.0.0.1:43848)
0807 07:05:22.179987 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:22.180080 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.1:32771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260807 07:05:22.240639 12562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:22.240157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:32771 (local address 127.0.0.1:43860)
0807 07:05:22.240377 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:22.240468 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.1:32771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260807 07:05:22.241662 12562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:22.241346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:32771 (local address 127.0.0.1:43872)
0807 07:05:22.241465 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:22.241547 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.1:32771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260807 07:05:22.259693 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 12174
I20260807 07:05:22.284299 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 12359
Aug 07 07:05:22 dist-test-slave-ncp9 krb5kdc[11828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.3@KRBTEST.COM
I20260807 07:05:22.298593 12565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:22.285002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38164 (local address 127.21.83.3:40235)
0807 07:05:22.285181 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0807 07:05:22.285187 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:05:22.285206 (+ 19us) server_negotiation.cc:407] Connection header received
0807 07:05:22.285397 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:05:22.285402 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:05:22.285467 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:05:22.285557 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:05:22.286896 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:22.287594 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:22.288826 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:22.289510 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:22.289627 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:05:22.293732 (+ 4105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:05:22.293746 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:05:22.293760 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:05:22.293793 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:05:22.296258 (+ 2465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:05:22.296858 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:05:22.296865 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:05:22.296867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:05:22.297048 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:05:22.297341 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:05:22.297344 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:05:22.297346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:05:22.297682 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:05:22.297896 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0807 07:05:22.298113 (+ 217us) server_negotiation.cc:299] Negotiation successful
0807 07:05:22.298279 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260807 07:05:22.299173 12537 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:05:22.322528 12562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:22.321919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:32771 (local address 127.0.0.1:43888)
0807 07:05:22.322180 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:22.322310 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.1:32771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260807 07:05:22.323673 12562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:22.323216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:32771 (local address 127.0.0.1:43898)
0807 07:05:22.323414 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:22.323504 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.1:32771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260807 07:05:22.389706 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 12359
I20260807 07:05:22.413367 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 11848
I20260807 07:05:22.414613 11946 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:05:22.526714 12562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:22.526174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:32771 (local address 127.0.0.1:43914)
0807 07:05:22.526406 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:22.526495 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.1:32771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260807 07:05:22.526885 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 11848
I20260807 07:05:42.553610 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:39001
--webserver_interface=127.21.83.62
--webserver_port=46785
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:38161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:39001
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:05:42.898656 12618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:05:42.898993 12618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:05:42.899133 12618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:05:42.906641 12618 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260807 07:05:42.906756 12618 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260807 07:05:42.906816 12618 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260807 07:05:42.906891 12618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:05:42.906967 12618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:05:42.907068 12618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:05:42.907141 12618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:05:42.907218 12618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:05:42.917382 12618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:38161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:39001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:39001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.83.62
--webserver_port=46785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:05:42.918733 12618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:05:42.920048 12618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:05:42.944581 12630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:05:42.944701 12627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:05:42.945246 12633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:05:42.945297 12618 server_base.cc:1034] running on GCE node
I20260807 07:05:42.946102 12618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:05:42.947518 12618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:05:42.948740 12618 hybrid_clock.cc:648] HybridClock initialized: now 1786086342948664 us; error 50 us; skew 500 ppm
I20260807 07:05:42.949152 12618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:05:42 dist-test-slave-ncp9 krb5kdc[11828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086342, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:05:42.955464 12618 init.cc:377] Logged in from keytab as kudu/127.21.83.62@KRBTEST.COM (short username kudu)
I20260807 07:05:42.957027 12618 webserver.cc:460] Webserver started at http://127.21.83.62:46785/ using document root <none> and password file <none>
I20260807 07:05:42.957742 12618 fs_manager.cc:359] Metadata directory not provided
I20260807 07:05:42.957844 12618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:05:42.963598 12618 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260807 07:05:42.966416 12643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:05:42.967859 12618 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 07:05:42.968052 12618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "dfc9b269350e44038e90a41e5b96c1f5"
format_stamp: "Formatted at 2026-08-07 07:05:19 on dist-test-slave-ncp9"
server_key: "229338d6ef901aca7a97ec9540716f87"
server_key_iv: "66fbb33c791cd5224cd12e9d5f734f93"
server_key_version: "encryptionkey@0"
I20260807 07:05:42.969156 12618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:05:42.985536 12618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:05:42.989382 12618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:05:42.989655 12618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:05:43.007727 12618 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.62:39001
I20260807 07:05:43.007746 12723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.62:39001 every 8 connection(s)
I20260807 07:05:43.009204 12618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:05:43.015861 12724 sys_catalog.cc:263] Verifying existing consensus state
I20260807 07:05:43.016681 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 12618
I20260807 07:05:43.017740 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.1:32771
--local_ip_for_outbound_sockets=127.21.83.1
--tserver_master_addrs=127.21.83.62:39001
--webserver_port=34403
--webserver_interface=127.21.83.1
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab
--principal=kudu/127.21.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:38161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 07:05:43.020323 12724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5: Bootstrap starting.
I20260807 07:05:43.042886 12724 log.cc:824] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5: Log is configured to *not* fsync() on all Append() calls
I20260807 07:05:43.055012 12724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 07:05:43.055536 12724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5: Bootstrap complete.
I20260807 07:05:43.064638 12724 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc9b269350e44038e90a41e5b96c1f5" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39001 } }
I20260807 07:05:43.065485 12724 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfc9b269350e44038e90a41e5b96c1f5, State: Initialized, Role: FOLLOWER
I20260807 07:05:43.066108 12724 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc9b269350e44038e90a41e5b96c1f5" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39001 } }
I20260807 07:05:43.066334 12724 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:05:43.066449 12724 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:05:43.066609 12724 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5 [term 1 FOLLOWER]: Advancing to term 2
I20260807 07:05:43.070343 12724 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc9b269350e44038e90a41e5b96c1f5" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39001 } }
I20260807 07:05:43.070834 12724 leader_election.cc:302] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfc9b269350e44038e90a41e5b96c1f5; no voters:
I20260807 07:05:43.071451 12724 leader_election.cc:288] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260807 07:05:43.071599 12730 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 07:05:43.072674 12730 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5 [term 2 LEADER]: Becoming Leader. State: Replica: dfc9b269350e44038e90a41e5b96c1f5, State: Running, Role: LEADER
I20260807 07:05:43.073422 12730 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc9b269350e44038e90a41e5b96c1f5" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39001 } }
I20260807 07:05:43.073699 12724 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:05:43.076467 12735 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfc9b269350e44038e90a41e5b96c1f5. Latest consensus state: current_term: 2 leader_uuid: "dfc9b269350e44038e90a41e5b96c1f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc9b269350e44038e90a41e5b96c1f5" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39001 } } }
I20260807 07:05:43.076491 12733 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "dfc9b269350e44038e90a41e5b96c1f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc9b269350e44038e90a41e5b96c1f5" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39001 } } }
I20260807 07:05:43.076907 12735 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5 [sys.catalog]: This master's current role is: LEADER
I20260807 07:05:43.076964 12733 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5 [sys.catalog]: This master's current role is: LEADER
I20260807 07:05:43.086195 12748 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:05:43.094445 12748 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:05:43.095952 12748 catalog_manager.cc:1259] Loaded cluster ID: 0f5b480469e24dcfbc8546b45d643a8d
I20260807 07:05:43.096103 12748 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:05:43.100252 12748 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:05:43.101944 12748 catalog_manager.cc:5879] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5: Loaded TSK: 0
I20260807 07:05:43.137861 12748 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5: Generated new TSK 1
I20260807 07:05:43.138247 12748 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 07:05:43.385646 12726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:05:43.385986 12726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:05:43.386139 12726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:05:43.393805 12726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:05:43.393950 12726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:05:43.394114 12726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.1
I20260807 07:05:43.404271 12726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:38161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab
--principal=kudu/127.21.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.1:32771
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.83.1
--webserver_port=34403
--tserver_master_addrs=127.21.83.62:39001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.1
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:05:43.405593 12726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:05:43.406841 12726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:05:43.421020 12768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:05:43.421221 12769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:05:43.421020 12771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:05:43.422204 12726 server_base.cc:1034] running on GCE node
I20260807 07:05:43.422813 12726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:05:43.423844 12726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:05:43.425055 12726 hybrid_clock.cc:648] HybridClock initialized: now 1786086343424998 us; error 49 us; skew 500 ppm
I20260807 07:05:43.425364 12726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:05:43 dist-test-slave-ncp9 krb5kdc[11828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086343, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:05:43.428403 12726 init.cc:377] Logged in from keytab as kudu/127.21.83.1@KRBTEST.COM (short username kudu)
I20260807 07:05:43.429996 12726 webserver.cc:460] Webserver started at http://127.21.83.1:34403/ using document root <none> and password file <none>
I20260807 07:05:43.430670 12726 fs_manager.cc:359] Metadata directory not provided
I20260807 07:05:43.430797 12726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:05:43.436266 12726 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260807 07:05:43.439494 12782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:05:43.440855 12726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 07:05:43.441075 12726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-0/wal
uuid: "faefba3fc057402dae24f9e78dfdf048"
format_stamp: "Formatted at 2026-08-07 07:05:20 on dist-test-slave-ncp9"
server_key: "70ce1b31c472ebdce137a2a6cf576add"
server_key_iv: "c907f32db2236ebaf31cab5bcc6d84c3"
server_key_version: "encryptionkey@0"
I20260807 07:05:43.442029 12726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:05:43.472855 12726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:05:43.477530 12726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:05:43.477813 12726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:05:43.478796 12726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:05:43.480571 12790 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260807 07:05:43.486042 12726 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260807 07:05:43.486162 12726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260807 07:05:43.486279 12726 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260807 07:05:43.489300 12726 ts_tablet_manager.cc:610] Registered 1 tablets
I20260807 07:05:43.489377 12726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20260807 07:05:43.489759 12790 tablet_bootstrap.cc:492] T 86ca1107ae564a6084fdb81af977419d P faefba3fc057402dae24f9e78dfdf048: Bootstrap starting.
I20260807 07:05:43.498180 12790 log.cc:824] T 86ca1107ae564a6084fdb81af977419d P faefba3fc057402dae24f9e78dfdf048: Log is configured to *not* fsync() on all Append() calls
I20260807 07:05:43.514290 12790 tablet_bootstrap.cc:492] T 86ca1107ae564a6084fdb81af977419d P faefba3fc057402dae24f9e78dfdf048: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 07:05:43.515082 12790 tablet_bootstrap.cc:492] T 86ca1107ae564a6084fdb81af977419d P faefba3fc057402dae24f9e78dfdf048: Bootstrap complete.
I20260807 07:05:43.515635 12790 ts_tablet_manager.cc:1397] T 86ca1107ae564a6084fdb81af977419d P faefba3fc057402dae24f9e78dfdf048: Time spent bootstrapping tablet: real 0.026s user 0.021s sys 0.001s
I20260807 07:05:43.523244 12726 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.1:32771
I20260807 07:05:43.523279 12948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.1:32771 every 8 connection(s)
I20260807 07:05:43.524825 12726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
I20260807 07:05:43.526587 12790 raft_consensus.cc:348] T 86ca1107ae564a6084fdb81af977419d P faefba3fc057402dae24f9e78dfdf048 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c62ec1636f34095a1fa40b04f80da9e" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 46577 } } peers { permanent_uuid: "faefba3fc057402dae24f9e78dfdf048" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 32771 } } peers { permanent_uuid: "0c43f4e5f9414fa6b99d9c0a7c0dd30c" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40235 } }
I20260807 07:05:43.527565 12790 raft_consensus.cc:729] T 86ca1107ae564a6084fdb81af977419d P faefba3fc057402dae24f9e78dfdf048 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: faefba3fc057402dae24f9e78dfdf048, State: Initialized, Role: FOLLOWER
I20260807 07:05:43.528282 12790 consensus_queue.cc:260] T 86ca1107ae564a6084fdb81af977419d P faefba3fc057402dae24f9e78dfdf048 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c62ec1636f34095a1fa40b04f80da9e" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 46577 } } peers { permanent_uuid: "faefba3fc057402dae24f9e78dfdf048" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 32771 } } peers { permanent_uuid: "0c43f4e5f9414fa6b99d9c0a7c0dd30c" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40235 } }
I20260807 07:05:43.530190 12790 ts_tablet_manager.cc:1428] T 86ca1107ae564a6084fdb81af977419d P faefba3fc057402dae24f9e78dfdf048: Time spent starting tablet: real 0.014s user 0.015s sys 0.001s
I20260807 07:05:43.530645 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 12726
I20260807 07:05:43.531464 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.2:46577
--local_ip_for_outbound_sockets=127.21.83.2
--tserver_master_addrs=127.21.83.62:39001
--webserver_port=45977
--webserver_interface=127.21.83.2
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab
--principal=kudu/127.21.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:38161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 07:05:43 dist-test-slave-ncp9 krb5kdc[11828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086343, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.1@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:05:43.548599 12952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:43.527761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.1:54943)
0807 07:05:43.528633 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:43.528691 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0807 07:05:43.529247 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:05:43.531320 (+ 2073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:05:43.531328 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:05:43.532011 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:05:43.532587 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:05:43.532605 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:05:43.534718 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:05:43.534723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:05:43.535962 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:05:43.535975 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:05:43.537272 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:05:43.537276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:05:43.537463 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:05:43.538199 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:05:43.538221 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:05:43.540509 (+ 2288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:05:43.544481 (+ 3972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:05:43.544490 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:05:43.544500 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:05:43.545093 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:05:43.545621 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:05:43.545625 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:05:43.545627 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:05:43.545802 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:05:43.546674 (+ 872us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:05:43.546679 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:05:43.546941 (+ 262us) client_negotiation.cc:769] Sending connection context
0807 07:05:43.547209 (+ 268us) client_negotiation.cc:240] Negotiation successful
0807 07:05:43.547551 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":384,"threads_started":1}
I20260807 07:05:43.548995 12953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:43.527927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:54943 (local address 127.21.83.62:39001)
0807 07:05:43.529915 (+ 1988us) server_negotiation.cc:206] Beginning negotiation
0807 07:05:43.529925 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 07:05:43.529965 (+ 40us) server_negotiation.cc:407] Connection header received
0807 07:05:43.530364 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:05:43.530383 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:05:43.530818 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:05:43.531205 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:05:43.532800 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:43.534536 (+ 1736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:43.536127 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:43.537069 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:43.537274 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:05:43.540787 (+ 3513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:05:43.540796 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:05:43.540812 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:05:43.540854 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:05:43.544292 (+ 3438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:05:43.545236 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:05:43.545245 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:05:43.545253 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:05:43.545465 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:05:43.545912 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:05:43.545916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:05:43.545920 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:05:43.546519 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:05:43.546688 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0807 07:05:43.547443 (+ 755us) server_negotiation.cc:299] Negotiation successful
0807 07:05:43.547825 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1818,"thread_start_us":1695,"threads_started":1}
I20260807 07:05:43.551412 12949 heartbeater.cc:347] Connected to a master server at 127.21.83.62:39001
I20260807 07:05:43.551713 12949 heartbeater.cc:464] Registering TS with master...
I20260807 07:05:43.552455 12949 heartbeater.cc:511] Master 127.21.83.62:39001 requested a full tablet report, sending...
I20260807 07:05:43.555619 12673 ts_manager.cc:194] Registered new tserver with Master: faefba3fc057402dae24f9e78dfdf048 (127.21.83.1:32771)
I20260807 07:05:43.560155 12673 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.1@KRBTEST.COM'} at 127.21.83.1:54943
I20260807 07:05:43.563956 12949 heartbeater.cc:503] Master 127.21.83.62:39001 was elected leader, sending a full tablet report...
Aug 07 07:05:43 dist-test-slave-ncp9 krb5kdc[11828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086342, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for kudu/127.21.83.1@KRBTEST.COM
I20260807 07:05:43.575428 12961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:43.558981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:32771 (local address 127.0.0.1:38988)
0807 07:05:43.559970 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:43.560002 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0807 07:05:43.560259 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:05:43.560972 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:05:43.560979 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:05:43.561877 (+ 898us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:05:43.562179 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:05:43.562203 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:05:43.563709 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:05:43.563714 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:05:43.564848 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:05:43.564866 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:05:43.566089 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:05:43.566096 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:05:43.566291 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:05:43.566894 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:05:43.566911 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:05:43.569060 (+ 2149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:05:43.572800 (+ 3740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:05:43.572810 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:05:43.572815 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:05:43.573314 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:05:43.573710 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:05:43.573713 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:05:43.573716 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:05:43.573861 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:05:43.574535 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:05:43.574542 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:05:43.574715 (+ 173us) client_negotiation.cc:769] Sending connection context
0807 07:05:43.574860 (+ 145us) client_negotiation.cc:240] Negotiation successful
0807 07:05:43.575101 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"thread_start_us":658,"threads_started":1}
I20260807 07:05:43.576074 12960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:43.558950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38988 (local address 127.21.83.1:32771)
0807 07:05:43.559513 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0807 07:05:43.559520 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:05:43.560109 (+ 589us) server_negotiation.cc:407] Connection header received
0807 07:05:43.560524 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:05:43.560533 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:05:43.560755 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:05:43.560925 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:05:43.562374 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:43.563490 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:43.565032 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:43.565855 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:43.566002 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:05:43.569276 (+ 3274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:05:43.569290 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:05:43.569306 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:05:43.569344 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:05:43.572688 (+ 3344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:05:43.573447 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:05:43.573455 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:05:43.573459 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:05:43.573610 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:05:43.573956 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:05:43.573962 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:05:43.573965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:05:43.574406 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:05:43.574595 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0807 07:05:43.575403 (+ 808us) server_negotiation.cc:299] Negotiation successful
0807 07:05:43.575694 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":261,"threads_started":1}
I20260807 07:05:43.577428 12856 tablet_service.cc:1514] Processing DeleteTablet for tablet 86ca1107ae564a6084fdb81af977419d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:05:21 UTC) from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:38988
I20260807 07:05:43.579834 12962 tablet_replica.cc:330] stopping tablet replica
I20260807 07:05:43.580178 12962 raft_consensus.cc:2229] T 86ca1107ae564a6084fdb81af977419d P faefba3fc057402dae24f9e78dfdf048 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:05:43.580554 12962 raft_consensus.cc:2258] T 86ca1107ae564a6084fdb81af977419d P faefba3fc057402dae24f9e78dfdf048 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:05:43.581791 12962 ts_tablet_manager.cc:1905] T 86ca1107ae564a6084fdb81af977419d P faefba3fc057402dae24f9e78dfdf048: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:05:43.588307 12962 ts_tablet_manager.cc:1918] T 86ca1107ae564a6084fdb81af977419d P faefba3fc057402dae24f9e78dfdf048: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:05:43.588431 12962 log.cc:1196] T 86ca1107ae564a6084fdb81af977419d P faefba3fc057402dae24f9e78dfdf048: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-0/wal/wals/86ca1107ae564a6084fdb81af977419d
I20260807 07:05:43.588868 12962 ts_tablet_manager.cc:1939] T 86ca1107ae564a6084fdb81af977419d P faefba3fc057402dae24f9e78dfdf048: Deleting consensus metadata
I20260807 07:05:43.590770 12647 catalog_manager.cc:4841] TS faefba3fc057402dae24f9e78dfdf048 (127.21.83.1:32771): tablet 86ca1107ae564a6084fdb81af977419d (table test-table [id=e30d1a6d5c4f484a8547bb824a8ae907]) successfully deleted
W20260807 07:05:43.869244 12955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:05:43.869504 12955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:05:43.869606 12955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:05:43.876791 12955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:05:43.876924 12955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:05:43.877072 12955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.2
I20260807 07:05:43.886626 12955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:38161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab
--principal=kudu/127.21.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.2:46577
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.83.2
--webserver_port=45977
--tserver_master_addrs=127.21.83.62:39001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.2
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:05:43.887832 12955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:05:43.889081 12955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:05:43.902251 12969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:05:43.902227 12976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:05:43.902179 12971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:05:43.902976 12955 server_base.cc:1034] running on GCE node
I20260807 07:05:43.903642 12955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:05:43.904768 12955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:05:43.906109 12955 hybrid_clock.cc:648] HybridClock initialized: now 1786086343906042 us; error 49 us; skew 500 ppm
I20260807 07:05:43.906459 12955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:05:43 dist-test-slave-ncp9 krb5kdc[11828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086343, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:05:43.909821 12955 init.cc:377] Logged in from keytab as kudu/127.21.83.2@KRBTEST.COM (short username kudu)
I20260807 07:05:43.912369 12955 webserver.cc:460] Webserver started at http://127.21.83.2:45977/ using document root <none> and password file <none>
I20260807 07:05:43.913223 12955 fs_manager.cc:359] Metadata directory not provided
I20260807 07:05:43.913374 12955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:05:43.918982 12955 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 07:05:43.921595 12985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:05:43.922821 12955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 07:05:43.922974 12955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-1/wal
uuid: "2c62ec1636f34095a1fa40b04f80da9e"
format_stamp: "Formatted at 2026-08-07 07:05:20 on dist-test-slave-ncp9"
server_key: "d50e3a0215607a0e4ffa9fdf98d2671b"
server_key_iv: "fd4e230fae2fb90a9775d32ffa2ccdb9"
server_key_version: "encryptionkey@0"
I20260807 07:05:43.924074 12955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:05:43.960594 12955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:05:43.964970 12955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:05:43.965291 12955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:05:43.966297 12955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:05:43.967737 12955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:05:43.967842 12955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:05:43.967962 12955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:05:43.968034 12955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:05:43.999177 12955 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.2:46577
I20260807 07:05:43.999221 13151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.2:46577 every 8 connection(s)
I20260807 07:05:44.000586 12955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
I20260807 07:05:44.010265 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 12955
I20260807 07:05:44.011052 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.3:40235
--local_ip_for_outbound_sockets=127.21.83.3
--tserver_master_addrs=127.21.83.62:39001
--webserver_port=41749
--webserver_interface=127.21.83.3
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab
--principal=kudu/127.21.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:38161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 07:05:44 dist-test-slave-ncp9 krb5kdc[11828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086343, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.2@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:05:44.024498 12953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:44.003174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:60233 (local address 127.21.83.62:39001)
0807 07:05:44.003368 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0807 07:05:44.003373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:05:44.004488 (+ 1115us) server_negotiation.cc:407] Connection header received
0807 07:05:44.006809 (+ 2321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:05:44.006814 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:05:44.006882 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:05:44.007005 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:05:44.009001 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:44.010082 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:44.012200 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:44.012876 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:44.013041 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:05:44.017379 (+ 4338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:05:44.017385 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:05:44.017389 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:05:44.017428 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:05:44.020666 (+ 3238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:05:44.021575 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:05:44.021581 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:05:44.021584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:05:44.021744 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:05:44.022341 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:05:44.022347 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:05:44.022350 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:05:44.022815 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:05:44.022983 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0807 07:05:44.023845 (+ 862us) server_negotiation.cc:299] Negotiation successful
0807 07:05:44.024097 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260807 07:05:44.025449 13156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:44.003635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.2:60233)
0807 07:05:44.004334 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:44.004370 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0807 07:05:44.006499 (+ 2129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:05:44.007253 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:05:44.007266 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:05:44.007995 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:05:44.008788 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:05:44.008811 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:05:44.010271 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:05:44.010276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:05:44.012052 (+ 1776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:05:44.012068 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:05:44.013125 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:05:44.013133 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:05:44.013379 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:05:44.014107 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:05:44.014141 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:05:44.017105 (+ 2964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:05:44.020805 (+ 3700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:05:44.020814 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:05:44.020827 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:05:44.021440 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:05:44.021856 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:05:44.021861 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:05:44.021865 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:05:44.022079 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:05:44.022981 (+ 902us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:05:44.022998 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:05:44.023392 (+ 394us) client_negotiation.cc:769] Sending connection context
0807 07:05:44.023704 (+ 312us) client_negotiation.cc:240] Negotiation successful
0807 07:05:44.024219 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":273,"threads_started":1}
I20260807 07:05:44.026863 13152 heartbeater.cc:347] Connected to a master server at 127.21.83.62:39001
I20260807 07:05:44.027242 13152 heartbeater.cc:464] Registering TS with master...
I20260807 07:05:44.028218 13152 heartbeater.cc:511] Master 127.21.83.62:39001 requested a full tablet report, sending...
I20260807 07:05:44.030058 12673 ts_manager.cc:194] Registered new tserver with Master: 2c62ec1636f34095a1fa40b04f80da9e (127.21.83.2:46577)
I20260807 07:05:44.030948 12673 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.2@KRBTEST.COM'} at 127.21.83.2:60233
W20260807 07:05:44.351883 13157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:05:44.352155 13157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:05:44.352298 13157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:05:44.359519 13157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:05:44.359637 13157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:05:44.359790 13157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.3
I20260807 07:05:44.369237 13157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:38161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab
--principal=kudu/127.21.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.3:40235
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.83.3
--webserver_port=41749
--tserver_master_addrs=127.21.83.62:39001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.3
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:05:44.370401 13157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:05:44.371629 13157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:05:44.384330 13162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:05:44.384264 13163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:05:44.384773 13167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:05:44.385236 13157 server_base.cc:1034] running on GCE node
I20260807 07:05:44.385797 13157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:05:44.386644 13157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:05:44.387835 13157 hybrid_clock.cc:648] HybridClock initialized: now 1786086344387797 us; error 31 us; skew 500 ppm
I20260807 07:05:44.388113 13157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:05:44 dist-test-slave-ncp9 krb5kdc[11828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086344, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:05:44.391095 13157 init.cc:377] Logged in from keytab as kudu/127.21.83.3@KRBTEST.COM (short username kudu)
I20260807 07:05:44.392371 13157 webserver.cc:460] Webserver started at http://127.21.83.3:41749/ using document root <none> and password file <none>
I20260807 07:05:44.393029 13157 fs_manager.cc:359] Metadata directory not provided
I20260807 07:05:44.393168 13157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:05:44.398139 13157 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260807 07:05:44.400520 13175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:05:44.401681 13157 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 07:05:44.401827 13157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-2/wal
uuid: "0c43f4e5f9414fa6b99d9c0a7c0dd30c"
format_stamp: "Formatted at 2026-08-07 07:05:21 on dist-test-slave-ncp9"
server_key: "044ff0c34bf96ac04c77383f7f3a477a"
server_key_iv: "f7f629e11f8723ad1e524c108555ca30"
server_key_version: "encryptionkey@0"
I20260807 07:05:44.402762 13157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:05:44.418768 13157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:05:44.423265 13157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:05:44.423664 13157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:05:44.424841 13157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:05:44.426390 13157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:05:44.426494 13157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:05:44.426595 13157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:05:44.426669 13157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:05:44.457391 13157 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.3:40235
I20260807 07:05:44.457460 13343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.3:40235 every 8 connection(s)
I20260807 07:05:44.458737 13157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
I20260807 07:05:44.466099 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 13157
Aug 07 07:05:44 dist-test-slave-ncp9 krb5kdc[11828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086344, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.3@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:05:44.479470 12953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:44.461130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:55609 (local address 127.21.83.62:39001)
0807 07:05:44.461295 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0807 07:05:44.461299 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:05:44.462402 (+ 1103us) server_negotiation.cc:407] Connection header received
0807 07:05:44.464324 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:05:44.464330 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:05:44.464411 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:05:44.464500 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:05:44.466214 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:44.467161 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:44.468851 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:44.469528 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:44.469654 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:05:44.473153 (+ 3499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:05:44.473157 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:05:44.473161 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:05:44.473194 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:05:44.476171 (+ 2977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:05:44.477099 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:05:44.477105 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:05:44.477108 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:05:44.477270 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:05:44.477733 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:05:44.477738 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:05:44.477742 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:05:44.478124 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:05:44.478296 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0807 07:05:44.478978 (+ 682us) server_negotiation.cc:299] Negotiation successful
0807 07:05:44.479218 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54,"spinlock_wait_cycles":896}
I20260807 07:05:44.480629 13349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:44.461494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.3:55609)
0807 07:05:44.462248 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:44.462287 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0807 07:05:44.464056 (+ 1769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:05:44.464758 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:05:44.464766 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:05:44.465319 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:05:44.466000 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:05:44.466027 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:05:44.467394 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:05:44.467400 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:05:44.468701 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:05:44.468714 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:05:44.469721 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:05:44.469726 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:05:44.469921 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:05:44.470521 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:05:44.470543 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:05:44.472965 (+ 2422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:05:44.476321 (+ 3356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:05:44.476328 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:05:44.476340 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:05:44.476946 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:05:44.477392 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:05:44.477396 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:05:44.477398 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:05:44.477593 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:05:44.478253 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:05:44.478259 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:05:44.478584 (+ 325us) client_negotiation.cc:769] Sending connection context
0807 07:05:44.478818 (+ 234us) client_negotiation.cc:240] Negotiation successful
0807 07:05:44.479255 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":335,"threads_started":1}
I20260807 07:05:44.482134 13352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:44.467022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58966 (local address 127.21.83.62:39001)
0807 07:05:44.467458 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0807 07:05:44.467464 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:05:44.467794 (+ 330us) server_negotiation.cc:407] Connection header received
0807 07:05:44.467959 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:05:44.467964 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:05:44.468024 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:05:44.468109 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:05:44.469992 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:44.470700 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:44.472233 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:44.472712 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:44.472807 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:05:44.476055 (+ 3248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:05:44.476060 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:05:44.476065 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:05:44.476110 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:05:44.479057 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:05:44.479932 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:05:44.479936 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:05:44.479938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:05:44.480051 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:05:44.480461 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:05:44.480464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:05:44.480466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:05:44.480719 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:05:44.481002 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0807 07:05:44.481545 (+ 543us) server_negotiation.cc:299] Negotiation successful
0807 07:05:44.481769 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"spinlock_wait_cycles":6528,"thread_start_us":236,"threads_started":1}
I20260807 07:05:44.482236 13345 heartbeater.cc:347] Connected to a master server at 127.21.83.62:39001
I20260807 07:05:44.482553 13345 heartbeater.cc:464] Registering TS with master...
I20260807 07:05:44.483340 13345 heartbeater.cc:511] Master 127.21.83.62:39001 requested a full tablet report, sending...
I20260807 07:05:44.484791 12673 ts_manager.cc:194] Registered new tserver with Master: 0c43f4e5f9414fa6b99d9c0a7c0dd30c (127.21.83.3:40235)
I20260807 07:05:44.485580 12673 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.3@KRBTEST.COM'} at 127.21.83.3:55609
I20260807 07:05:44.497274 21836 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 07:05:44.503695 13352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:44.498231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58974 (local address 127.21.83.62:39001)
0807 07:05:44.498374 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0807 07:05:44.498380 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:05:44.498838 (+ 458us) server_negotiation.cc:407] Connection header received
0807 07:05:44.499064 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:05:44.499067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:05:44.499219 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:05:44.499342 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:05:44.499715 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:44.500414 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:44.501943 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:44.502388 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:44.502481 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:05:44.503007 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:05:44.503209 (+ 202us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0807 07:05:44.503486 (+ 277us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58974: token expired
Metrics: {"server-negotiator.queue_time_us":35}
W20260807 07:05:44.503818 13352 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58974: token expired
W20260807 07:05:44.504364 12552 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.21.83.62:39001)
I20260807 07:05:44.516203 13352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:44.504776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58988 (local address 127.21.83.62:39001)
0807 07:05:44.504948 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0807 07:05:44.504953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:05:44.504970 (+ 17us) server_negotiation.cc:407] Connection header received
0807 07:05:44.505097 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:05:44.505100 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:05:44.505163 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:05:44.505234 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:05:44.506616 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:44.507301 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:44.508634 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:44.509049 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:44.509139 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:05:44.511691 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:05:44.511696 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:05:44.511699 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:05:44.511729 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:05:44.514123 (+ 2394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:05:44.514705 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:05:44.514708 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:05:44.514711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:05:44.514851 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:05:44.515119 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:05:44.515122 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:05:44.515124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:05:44.515366 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:05:44.515539 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0807 07:05:44.515758 (+ 219us) server_negotiation.cc:299] Negotiation successful
0807 07:05:44.515912 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260807 07:05:44.519466 12673 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58988:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 07:05:44.527004 12673 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 07:05:44.541090 12856 tablet_service.cc:1467] Processing CreateTablet for tablet 863dd112e42f46dd9759a7fb11821ef1 (DEFAULT_TABLE table=test-table [id=91dbe99a058d4cd3a31a6e1a62d44640]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:05:44.542071 12856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 863dd112e42f46dd9759a7fb11821ef1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:05:44.547933 13364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:44.538150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.3:40235 (local address 127.0.0.1:51328)
0807 07:05:44.539011 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:44.539037 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 07:05:44.539179 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:05:44.540533 (+ 1354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:05:44.540538 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:05:44.540563 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:05:44.540984 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:05:44.541001 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:05:44.542779 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:05:44.542784 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:05:44.545219 (+ 2435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:05:44.545236 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:05:44.547163 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:05:44.547169 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:05:44.547333 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:05:44.547363 (+ 30us) client_negotiation.cc:769] Sending connection context
0807 07:05:44.547435 (+ 72us) client_negotiation.cc:240] Negotiation successful
0807 07:05:44.547568 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"thread_start_us":575,"threads_started":1}
I20260807 07:05:44.547933 13369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:44.539075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48724 (local address 127.21.83.2:46577)
0807 07:05:44.540102 (+ 1027us) server_negotiation.cc:206] Beginning negotiation
0807 07:05:44.540110 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 07:05:44.540133 (+ 23us) server_negotiation.cc:407] Connection header received
0807 07:05:44.540207 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:05:44.540212 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:05:44.540419 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:05:44.540628 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:05:44.541566 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:44.543040 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:44.545601 (+ 2561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:44.546622 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:44.546744 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:05:44.546967 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0807 07:05:44.547456 (+ 489us) server_negotiation.cc:299] Negotiation successful
0807 07:05:44.547638 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":895,"thread_start_us":769,"threads_started":1}
I20260807 07:05:44.548650 13365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:44.538507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.2:46577 (local address 127.0.0.1:48724)
0807 07:05:44.539330 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:44.539352 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 07:05:44.539451 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:05:44.540930 (+ 1479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:05:44.540935 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:05:44.540968 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:05:44.541357 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:05:44.541372 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:05:44.543198 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:05:44.543203 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:05:44.545442 (+ 2239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:05:44.545453 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:05:44.546830 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:05:44.546837 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:05:44.547027 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:05:44.547096 (+ 69us) client_negotiation.cc:769] Sending connection context
0807 07:05:44.547193 (+ 97us) client_negotiation.cc:240] Negotiation successful
0807 07:05:44.548236 (+ 1043us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":1044,"threads_started":1}
I20260807 07:05:44.549206 13368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:44.538418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51328 (local address 127.21.83.3:40235)
0807 07:05:44.539827 (+ 1409us) server_negotiation.cc:206] Beginning negotiation
0807 07:05:44.539836 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 07:05:44.539872 (+ 36us) server_negotiation.cc:407] Connection header received
0807 07:05:44.539984 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:05:44.539993 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:05:44.540277 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:05:44.541067 (+ 790us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:05:44.541364 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:44.542623 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:44.545420 (+ 2797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:44.546880 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:44.547010 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:05:44.548600 (+ 1590us) server_negotiation.cc:1035] Waiting for connection context
0807 07:05:44.548695 (+ 95us) server_negotiation.cc:299] Negotiation successful
0807 07:05:44.548862 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1227,"thread_start_us":1036,"threads_started":1}
I20260807 07:05:44.550385 13370 tablet_bootstrap.cc:492] T 863dd112e42f46dd9759a7fb11821ef1 P faefba3fc057402dae24f9e78dfdf048: Bootstrap starting.
I20260807 07:05:44.552335 13370 tablet_bootstrap.cc:654] T 863dd112e42f46dd9759a7fb11821ef1 P faefba3fc057402dae24f9e78dfdf048: Neither blocks nor log segments found. Creating new log.
I20260807 07:05:44.552274 13057 tablet_service.cc:1467] Processing CreateTablet for tablet 863dd112e42f46dd9759a7fb11821ef1 (DEFAULT_TABLE table=test-table [id=91dbe99a058d4cd3a31a6e1a62d44640]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:05:44.552704 13246 tablet_service.cc:1467] Processing CreateTablet for tablet 863dd112e42f46dd9759a7fb11821ef1 (DEFAULT_TABLE table=test-table [id=91dbe99a058d4cd3a31a6e1a62d44640]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:05:44.553944 13057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 863dd112e42f46dd9759a7fb11821ef1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:05:44.554068 13246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 863dd112e42f46dd9759a7fb11821ef1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:05:44.554926 13370 tablet_bootstrap.cc:492] T 863dd112e42f46dd9759a7fb11821ef1 P faefba3fc057402dae24f9e78dfdf048: No bootstrap required, opened a new log
I20260807 07:05:44.555086 13370 ts_tablet_manager.cc:1397] T 863dd112e42f46dd9759a7fb11821ef1 P faefba3fc057402dae24f9e78dfdf048: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260807 07:05:44.555965 13370 raft_consensus.cc:348] T 863dd112e42f46dd9759a7fb11821ef1 P faefba3fc057402dae24f9e78dfdf048 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c43f4e5f9414fa6b99d9c0a7c0dd30c" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40235 } } peers { permanent_uuid: "2c62ec1636f34095a1fa40b04f80da9e" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 46577 } } peers { permanent_uuid: "faefba3fc057402dae24f9e78dfdf048" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 32771 } }
I20260807 07:05:44.556218 13370 raft_consensus.cc:374] T 863dd112e42f46dd9759a7fb11821ef1 P faefba3fc057402dae24f9e78dfdf048 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:05:44.556375 13370 raft_consensus.cc:729] T 863dd112e42f46dd9759a7fb11821ef1 P faefba3fc057402dae24f9e78dfdf048 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faefba3fc057402dae24f9e78dfdf048, State: Initialized, Role: FOLLOWER
I20260807 07:05:44.556672 13370 consensus_queue.cc:260] T 863dd112e42f46dd9759a7fb11821ef1 P faefba3fc057402dae24f9e78dfdf048 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c43f4e5f9414fa6b99d9c0a7c0dd30c" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40235 } } peers { permanent_uuid: "2c62ec1636f34095a1fa40b04f80da9e" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 46577 } } peers { permanent_uuid: "faefba3fc057402dae24f9e78dfdf048" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 32771 } }
I20260807 07:05:44.560065 13370 ts_tablet_manager.cc:1428] T 863dd112e42f46dd9759a7fb11821ef1 P faefba3fc057402dae24f9e78dfdf048: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260807 07:05:44.567253 13373 tablet_bootstrap.cc:492] T 863dd112e42f46dd9759a7fb11821ef1 P 2c62ec1636f34095a1fa40b04f80da9e: Bootstrap starting.
I20260807 07:05:44.567353 13377 tablet_bootstrap.cc:492] T 863dd112e42f46dd9759a7fb11821ef1 P 0c43f4e5f9414fa6b99d9c0a7c0dd30c: Bootstrap starting.
I20260807 07:05:44.570163 13373 tablet_bootstrap.cc:654] T 863dd112e42f46dd9759a7fb11821ef1 P 2c62ec1636f34095a1fa40b04f80da9e: Neither blocks nor log segments found. Creating new log.
I20260807 07:05:44.570199 13377 tablet_bootstrap.cc:654] T 863dd112e42f46dd9759a7fb11821ef1 P 0c43f4e5f9414fa6b99d9c0a7c0dd30c: Neither blocks nor log segments found. Creating new log.
I20260807 07:05:44.571288 13377 log.cc:824] T 863dd112e42f46dd9759a7fb11821ef1 P 0c43f4e5f9414fa6b99d9c0a7c0dd30c: Log is configured to *not* fsync() on all Append() calls
I20260807 07:05:44.571291 13373 log.cc:824] T 863dd112e42f46dd9759a7fb11821ef1 P 2c62ec1636f34095a1fa40b04f80da9e: Log is configured to *not* fsync() on all Append() calls
I20260807 07:05:44.574354 13377 tablet_bootstrap.cc:492] T 863dd112e42f46dd9759a7fb11821ef1 P 0c43f4e5f9414fa6b99d9c0a7c0dd30c: No bootstrap required, opened a new log
I20260807 07:05:44.574352 13373 tablet_bootstrap.cc:492] T 863dd112e42f46dd9759a7fb11821ef1 P 2c62ec1636f34095a1fa40b04f80da9e: No bootstrap required, opened a new log
I20260807 07:05:44.574625 13373 ts_tablet_manager.cc:1397] T 863dd112e42f46dd9759a7fb11821ef1 P 2c62ec1636f34095a1fa40b04f80da9e: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260807 07:05:44.574622 13377 ts_tablet_manager.cc:1397] T 863dd112e42f46dd9759a7fb11821ef1 P 0c43f4e5f9414fa6b99d9c0a7c0dd30c: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260807 07:05:44.585224 13377 raft_consensus.cc:348] T 863dd112e42f46dd9759a7fb11821ef1 P 0c43f4e5f9414fa6b99d9c0a7c0dd30c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c43f4e5f9414fa6b99d9c0a7c0dd30c" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40235 } } peers { permanent_uuid: "2c62ec1636f34095a1fa40b04f80da9e" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 46577 } } peers { permanent_uuid: "faefba3fc057402dae24f9e78dfdf048" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 32771 } }
I20260807 07:05:44.585224 13373 raft_consensus.cc:348] T 863dd112e42f46dd9759a7fb11821ef1 P 2c62ec1636f34095a1fa40b04f80da9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c43f4e5f9414fa6b99d9c0a7c0dd30c" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40235 } } peers { permanent_uuid: "2c62ec1636f34095a1fa40b04f80da9e" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 46577 } } peers { permanent_uuid: "faefba3fc057402dae24f9e78dfdf048" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 32771 } }
I20260807 07:05:44.585721 13377 raft_consensus.cc:374] T 863dd112e42f46dd9759a7fb11821ef1 P 0c43f4e5f9414fa6b99d9c0a7c0dd30c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:05:44.585729 13373 raft_consensus.cc:374] T 863dd112e42f46dd9759a7fb11821ef1 P 2c62ec1636f34095a1fa40b04f80da9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:05:44.585836 13373 raft_consensus.cc:729] T 863dd112e42f46dd9759a7fb11821ef1 P 2c62ec1636f34095a1fa40b04f80da9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c62ec1636f34095a1fa40b04f80da9e, State: Initialized, Role: FOLLOWER
I20260807 07:05:44.585836 13377 raft_consensus.cc:729] T 863dd112e42f46dd9759a7fb11821ef1 P 0c43f4e5f9414fa6b99d9c0a7c0dd30c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c43f4e5f9414fa6b99d9c0a7c0dd30c, State: Initialized, Role: FOLLOWER
I20260807 07:05:44.586504 13377 consensus_queue.cc:260] T 863dd112e42f46dd9759a7fb11821ef1 P 0c43f4e5f9414fa6b99d9c0a7c0dd30c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c43f4e5f9414fa6b99d9c0a7c0dd30c" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40235 } } peers { permanent_uuid: "2c62ec1636f34095a1fa40b04f80da9e" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 46577 } } peers { permanent_uuid: "faefba3fc057402dae24f9e78dfdf048" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 32771 } }
I20260807 07:05:44.586505 13373 consensus_queue.cc:260] T 863dd112e42f46dd9759a7fb11821ef1 P 2c62ec1636f34095a1fa40b04f80da9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c43f4e5f9414fa6b99d9c0a7c0dd30c" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40235 } } peers { permanent_uuid: "2c62ec1636f34095a1fa40b04f80da9e" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 46577 } } peers { permanent_uuid: "faefba3fc057402dae24f9e78dfdf048" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 32771 } }
I20260807 07:05:44.587697 13345 heartbeater.cc:503] Master 127.21.83.62:39001 was elected leader, sending a full tablet report...
I20260807 07:05:44.588193 13152 heartbeater.cc:503] Master 127.21.83.62:39001 was elected leader, sending a full tablet report...
I20260807 07:05:44.588516 13377 ts_tablet_manager.cc:1428] T 863dd112e42f46dd9759a7fb11821ef1 P 0c43f4e5f9414fa6b99d9c0a7c0dd30c: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260807 07:05:44.588729 13373 ts_tablet_manager.cc:1428] T 863dd112e42f46dd9759a7fb11821ef1 P 2c62ec1636f34095a1fa40b04f80da9e: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260807 07:05:44.607923 13382 raft_consensus.cc:482] T 863dd112e42f46dd9759a7fb11821ef1 P 2c62ec1636f34095a1fa40b04f80da9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:05:44.608227 13382 raft_consensus.cc:504] T 863dd112e42f46dd9759a7fb11821ef1 P 2c62ec1636f34095a1fa40b04f80da9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c43f4e5f9414fa6b99d9c0a7c0dd30c" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40235 } } peers { permanent_uuid: "2c62ec1636f34095a1fa40b04f80da9e" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 46577 } } peers { permanent_uuid: "faefba3fc057402dae24f9e78dfdf048" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 32771 } }
I20260807 07:05:44.609836 13382 leader_election.cc:288] T 863dd112e42f46dd9759a7fb11821ef1 P 2c62ec1636f34095a1fa40b04f80da9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c43f4e5f9414fa6b99d9c0a7c0dd30c (127.21.83.3:40235), faefba3fc057402dae24f9e78dfdf048 (127.21.83.1:32771)
I20260807 07:05:44.617017 13368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:44.610175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:48483 (local address 127.21.83.3:40235)
0807 07:05:44.610393 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0807 07:05:44.610400 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:05:44.611122 (+ 722us) server_negotiation.cc:407] Connection header received
0807 07:05:44.611433 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:05:44.611438 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:05:44.611532 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:05:44.611597 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:05:44.612463 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:44.613268 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:44.615181 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:44.616108 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:44.616217 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:05:44.616317 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0807 07:05:44.616632 (+ 315us) server_negotiation.cc:299] Negotiation successful
0807 07:05:44.616755 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 07:05:44.617079 13386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:44.610659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:50517 (local address 127.21.83.1:32771)
0807 07:05:44.611645 (+ 986us) server_negotiation.cc:206] Beginning negotiation
0807 07:05:44.611651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:05:44.611668 (+ 17us) server_negotiation.cc:407] Connection header received
0807 07:05:44.611738 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:05:44.611742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:05:44.611835 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:05:44.611972 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:05:44.612647 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:44.613463 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:44.615179 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:44.616009 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:44.616125 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:05:44.616357 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0807 07:05:44.616625 (+ 268us) server_negotiation.cc:299] Negotiation successful
0807 07:05:44.616756 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":856,"thread_start_us":722,"threads_started":1}
I20260807 07:05:44.617172 13383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:44.610201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.3:40235 (local address 127.21.83.2:48483)
0807 07:05:44.611014 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:44.611036 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 07:05:44.611250 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:05:44.611802 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:05:44.611806 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:05:44.611833 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:05:44.612297 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:05:44.612312 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:05:44.613403 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:05:44.613409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:05:44.615020 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:05:44.615032 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:05:44.616317 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:05:44.616323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:05:44.616463 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:05:44.616503 (+ 40us) client_negotiation.cc:769] Sending connection context
0807 07:05:44.616581 (+ 78us) client_negotiation.cc:240] Negotiation successful
0807 07:05:44.616742 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"mutex_wait_us":162,"thread_start_us":442,"threads_started":1}
I20260807 07:05:44.617316 13385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:44.610289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:32771 (local address 127.21.83.2:50517)
0807 07:05:44.611377 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0807 07:05:44.611396 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 07:05:44.611514 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:05:44.611979 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:05:44.611982 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:05:44.612004 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:05:44.612483 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:05:44.612495 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:05:44.613607 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:05:44.613614 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:05:44.615033 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:05:44.615045 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:05:44.616226 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:05:44.616232 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:05:44.616413 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:05:44.616503 (+ 90us) client_negotiation.cc:769] Sending connection context
0807 07:05:44.616581 (+ 78us) client_negotiation.cc:240] Negotiation successful
0807 07:05:44.616733 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":942,"mutex_wait_us":93,"thread_start_us":717,"threads_started":1}
I20260807 07:05:44.618073 13277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "863dd112e42f46dd9759a7fb11821ef1" candidate_uuid: "2c62ec1636f34095a1fa40b04f80da9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c43f4e5f9414fa6b99d9c0a7c0dd30c" is_pre_election: true
I20260807 07:05:44.618112 12884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "863dd112e42f46dd9759a7fb11821ef1" candidate_uuid: "2c62ec1636f34095a1fa40b04f80da9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faefba3fc057402dae24f9e78dfdf048" is_pre_election: true
I20260807 07:05:44.618563 13277 raft_consensus.cc:2454] T 863dd112e42f46dd9759a7fb11821ef1 P 0c43f4e5f9414fa6b99d9c0a7c0dd30c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c62ec1636f34095a1fa40b04f80da9e in term 0.
I20260807 07:05:44.618615 12884 raft_consensus.cc:2454] T 863dd112e42f46dd9759a7fb11821ef1 P faefba3fc057402dae24f9e78dfdf048 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c62ec1636f34095a1fa40b04f80da9e in term 0.
I20260807 07:05:44.619244 12987 leader_election.cc:302] T 863dd112e42f46dd9759a7fb11821ef1 P 2c62ec1636f34095a1fa40b04f80da9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c62ec1636f34095a1fa40b04f80da9e, faefba3fc057402dae24f9e78dfdf048; no voters:
I20260807 07:05:44.619643 13382 raft_consensus.cc:2781] T 863dd112e42f46dd9759a7fb11821ef1 P 2c62ec1636f34095a1fa40b04f80da9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 07:05:44.619776 13382 raft_consensus.cc:482] T 863dd112e42f46dd9759a7fb11821ef1 P 2c62ec1636f34095a1fa40b04f80da9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:05:44.619874 13382 raft_consensus.cc:3037] T 863dd112e42f46dd9759a7fb11821ef1 P 2c62ec1636f34095a1fa40b04f80da9e [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:05:44.622078 13382 raft_consensus.cc:504] T 863dd112e42f46dd9759a7fb11821ef1 P 2c62ec1636f34095a1fa40b04f80da9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c43f4e5f9414fa6b99d9c0a7c0dd30c" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40235 } } peers { permanent_uuid: "2c62ec1636f34095a1fa40b04f80da9e" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 46577 } } peers { permanent_uuid: "faefba3fc057402dae24f9e78dfdf048" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 32771 } }
I20260807 07:05:44.622640 13382 leader_election.cc:288] T 863dd112e42f46dd9759a7fb11821ef1 P 2c62ec1636f34095a1fa40b04f80da9e [CANDIDATE]: Term 1 election: Requested vote from peers 0c43f4e5f9414fa6b99d9c0a7c0dd30c (127.21.83.3:40235), faefba3fc057402dae24f9e78dfdf048 (127.21.83.1:32771)
I20260807 07:05:44.623044 13277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "863dd112e42f46dd9759a7fb11821ef1" candidate_uuid: "2c62ec1636f34095a1fa40b04f80da9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c43f4e5f9414fa6b99d9c0a7c0dd30c"
I20260807 07:05:44.623086 12884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "863dd112e42f46dd9759a7fb11821ef1" candidate_uuid: "2c62ec1636f34095a1fa40b04f80da9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faefba3fc057402dae24f9e78dfdf048"
I20260807 07:05:44.623210 13277 raft_consensus.cc:3037] T 863dd112e42f46dd9759a7fb11821ef1 P 0c43f4e5f9414fa6b99d9c0a7c0dd30c [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:05:44.623253 12884 raft_consensus.cc:3037] T 863dd112e42f46dd9759a7fb11821ef1 P faefba3fc057402dae24f9e78dfdf048 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:05:44.625422 12884 raft_consensus.cc:2454] T 863dd112e42f46dd9759a7fb11821ef1 P faefba3fc057402dae24f9e78dfdf048 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c62ec1636f34095a1fa40b04f80da9e in term 1.
I20260807 07:05:44.625442 13277 raft_consensus.cc:2454] T 863dd112e42f46dd9759a7fb11821ef1 P 0c43f4e5f9414fa6b99d9c0a7c0dd30c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c62ec1636f34095a1fa40b04f80da9e in term 1.
I20260807 07:05:44.625795 12986 leader_election.cc:302] T 863dd112e42f46dd9759a7fb11821ef1 P 2c62ec1636f34095a1fa40b04f80da9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c43f4e5f9414fa6b99d9c0a7c0dd30c, 2c62ec1636f34095a1fa40b04f80da9e; no voters:
I20260807 07:05:44.626080 13382 raft_consensus.cc:2781] T 863dd112e42f46dd9759a7fb11821ef1 P 2c62ec1636f34095a1fa40b04f80da9e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:05:44.626495 13382 raft_consensus.cc:686] T 863dd112e42f46dd9759a7fb11821ef1 P 2c62ec1636f34095a1fa40b04f80da9e [term 1 LEADER]: Becoming Leader. State: Replica: 2c62ec1636f34095a1fa40b04f80da9e, State: Running, Role: LEADER
I20260807 07:05:44.626909 13382 consensus_queue.cc:237] T 863dd112e42f46dd9759a7fb11821ef1 P 2c62ec1636f34095a1fa40b04f80da9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c43f4e5f9414fa6b99d9c0a7c0dd30c" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40235 } } peers { permanent_uuid: "2c62ec1636f34095a1fa40b04f80da9e" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 46577 } } peers { permanent_uuid: "faefba3fc057402dae24f9e78dfdf048" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 32771 } }
I20260807 07:05:44.631304 12671 catalog_manager.cc:5495] T 863dd112e42f46dd9759a7fb11821ef1 P 2c62ec1636f34095a1fa40b04f80da9e reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c62ec1636f34095a1fa40b04f80da9e (127.21.83.2). New cstate: current_term: 1 leader_uuid: "2c62ec1636f34095a1fa40b04f80da9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c43f4e5f9414fa6b99d9c0a7c0dd30c" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c62ec1636f34095a1fa40b04f80da9e" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 46577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "faefba3fc057402dae24f9e78dfdf048" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 32771 } health_report { overall_health: UNKNOWN } } }
I20260807 07:05:44.649748 13369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:44.644039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48726 (local address 127.21.83.2:46577)
0807 07:05:44.644234 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0807 07:05:44.644240 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:05:44.644303 (+ 63us) server_negotiation.cc:407] Connection header received
0807 07:05:44.644542 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:05:44.644545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:05:44.644621 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:05:44.644758 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:05:44.645210 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:44.645930 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:44.647503 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:44.647961 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:44.648078 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:05:44.648629 (+ 551us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:05:44.649123 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:05:44.649259 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0807 07:05:44.649382 (+ 123us) server_negotiation.cc:299] Negotiation successful
0807 07:05:44.649472 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 07:05:44.657089 12884 raft_consensus.cc:1261] T 863dd112e42f46dd9759a7fb11821ef1 P faefba3fc057402dae24f9e78dfdf048 [term 1 FOLLOWER]: Refusing update from remote peer 2c62ec1636f34095a1fa40b04f80da9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:05:44.657140 13277 raft_consensus.cc:1261] T 863dd112e42f46dd9759a7fb11821ef1 P 0c43f4e5f9414fa6b99d9c0a7c0dd30c [term 1 FOLLOWER]: Refusing update from remote peer 2c62ec1636f34095a1fa40b04f80da9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:05:44.657910 13389 consensus_queue.cc:1035] T 863dd112e42f46dd9759a7fb11821ef1 P 2c62ec1636f34095a1fa40b04f80da9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "faefba3fc057402dae24f9e78dfdf048" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 32771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:05:44.658210 13382 consensus_queue.cc:1035] T 863dd112e42f46dd9759a7fb11821ef1 P 2c62ec1636f34095a1fa40b04f80da9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c43f4e5f9414fa6b99d9c0a7c0dd30c" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 40235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:05:44.668418 13392 mvcc.cc:205] Tried to move back new op lower bound from 7315809667706351616 to 7315809667595382784. Current Snapshot: MvccSnapshot[applied={T|T < 7315809667706351616}]
I20260807 07:05:44.673949 13397 mvcc.cc:205] Tried to move back new op lower bound from 7315809667706351616 to 7315809667595382784. Current Snapshot: MvccSnapshot[applied={T|T < 7315809667706351616}]
I20260807 07:05:44.681084 12673 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:05:44.681370 12673 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:05:44.684805 12673 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5: Sending DeleteTablet for 3 replicas of tablet 863dd112e42f46dd9759a7fb11821ef1
I20260807 07:05:44.685745 13246 tablet_service.cc:1514] Processing DeleteTablet for tablet 863dd112e42f46dd9759a7fb11821ef1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:05:44 UTC) from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:51328
I20260807 07:05:44.685933 13057 tablet_service.cc:1514] Processing DeleteTablet for tablet 863dd112e42f46dd9759a7fb11821ef1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:05:44 UTC) from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:48724
I20260807 07:05:44.685979 12856 tablet_service.cc:1514] Processing DeleteTablet for tablet 863dd112e42f46dd9759a7fb11821ef1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:05:44 UTC) from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:38988
I20260807 07:05:44.686653 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 12726
I20260807 07:05:44.697281 13410 tablet_replica.cc:330] T 863dd112e42f46dd9759a7fb11821ef1 P 0c43f4e5f9414fa6b99d9c0a7c0dd30c: stopping tablet replica
I20260807 07:05:44.697721 13410 raft_consensus.cc:2229] T 863dd112e42f46dd9759a7fb11821ef1 P 0c43f4e5f9414fa6b99d9c0a7c0dd30c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:05:44.698225 13411 tablet_replica.cc:330] T 863dd112e42f46dd9759a7fb11821ef1 P faefba3fc057402dae24f9e78dfdf048: stopping tablet replica
I20260807 07:05:44.698395 13411 raft_consensus.cc:2229] T 863dd112e42f46dd9759a7fb11821ef1 P faefba3fc057402dae24f9e78dfdf048 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:05:44.698659 13411 raft_consensus.cc:2258] T 863dd112e42f46dd9759a7fb11821ef1 P faefba3fc057402dae24f9e78dfdf048 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:05:44.699169 13410 raft_consensus.cc:2258] T 863dd112e42f46dd9759a7fb11821ef1 P 0c43f4e5f9414fa6b99d9c0a7c0dd30c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:05:44.699957 13413 tablet_replica.cc:330] T 863dd112e42f46dd9759a7fb11821ef1 P 2c62ec1636f34095a1fa40b04f80da9e: stopping tablet replica
I20260807 07:05:44.700279 13413 raft_consensus.cc:2229] T 863dd112e42f46dd9759a7fb11821ef1 P 2c62ec1636f34095a1fa40b04f80da9e [term 1 LEADER]: Raft consensus shutting down.
I20260807 07:05:44.700776 13413 raft_consensus.cc:2258] T 863dd112e42f46dd9759a7fb11821ef1 P 2c62ec1636f34095a1fa40b04f80da9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:05:44.701436 13411 ts_tablet_manager.cc:1905] T 863dd112e42f46dd9759a7fb11821ef1 P faefba3fc057402dae24f9e78dfdf048: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:05:44.702546 13410 ts_tablet_manager.cc:1905] T 863dd112e42f46dd9759a7fb11821ef1 P 0c43f4e5f9414fa6b99d9c0a7c0dd30c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:05:44.706688 13411 ts_tablet_manager.cc:1918] T 863dd112e42f46dd9759a7fb11821ef1 P faefba3fc057402dae24f9e78dfdf048: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:05:44.706826 13411 log.cc:1196] T 863dd112e42f46dd9759a7fb11821ef1 P faefba3fc057402dae24f9e78dfdf048: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-0/wal/wals/863dd112e42f46dd9759a7fb11821ef1
I20260807 07:05:44.707271 13411 ts_tablet_manager.cc:1939] T 863dd112e42f46dd9759a7fb11821ef1 P faefba3fc057402dae24f9e78dfdf048: Deleting consensus metadata
I20260807 07:05:44.707690 13386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:44.688211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39004 (local address 127.21.83.1:32771)
0807 07:05:44.688396 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0807 07:05:44.688402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:05:44.688420 (+ 18us) server_negotiation.cc:407] Connection header received
0807 07:05:44.688475 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:05:44.688480 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:05:44.688555 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:05:44.688646 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:05:44.690696 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:44.691863 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:44.693940 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:44.694611 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:44.694725 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:05:44.699850 (+ 5125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:05:44.699857 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:05:44.699861 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:05:44.699899 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:05:44.703245 (+ 3346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:05:44.704073 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:05:44.704079 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:05:44.704082 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:05:44.704251 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:05:44.704746 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:05:44.704751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:05:44.704754 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:05:44.705181 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:05:44.705879 (+ 698us) server_negotiation.cc:1035] Waiting for connection context
0807 07:05:44.707098 (+ 1219us) server_negotiation.cc:299] Negotiation successful
0807 07:05:44.707358 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 07:05:44.708424 12939 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:05:44.708449 12647 catalog_manager.cc:4841] TS faefba3fc057402dae24f9e78dfdf048 (127.21.83.1:32771): tablet 863dd112e42f46dd9759a7fb11821ef1 (table test-table [id=91dbe99a058d4cd3a31a6e1a62d44640]) successfully deleted
I20260807 07:05:44.710350 13413 ts_tablet_manager.cc:1905] T 863dd112e42f46dd9759a7fb11821ef1 P 2c62ec1636f34095a1fa40b04f80da9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:05:44.710407 13410 ts_tablet_manager.cc:1918] T 863dd112e42f46dd9759a7fb11821ef1 P 0c43f4e5f9414fa6b99d9c0a7c0dd30c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:05:44.711174 13410 log.cc:1196] T 863dd112e42f46dd9759a7fb11821ef1 P 0c43f4e5f9414fa6b99d9c0a7c0dd30c: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-2/wal/wals/863dd112e42f46dd9759a7fb11821ef1
I20260807 07:05:44.711818 13410 ts_tablet_manager.cc:1939] T 863dd112e42f46dd9759a7fb11821ef1 P 0c43f4e5f9414fa6b99d9c0a7c0dd30c: Deleting consensus metadata
I20260807 07:05:44.713690 12644 catalog_manager.cc:4841] TS 0c43f4e5f9414fa6b99d9c0a7c0dd30c (127.21.83.3:40235): tablet 863dd112e42f46dd9759a7fb11821ef1 (table test-table [id=91dbe99a058d4cd3a31a6e1a62d44640]) successfully deleted
I20260807 07:05:44.718200 13413 ts_tablet_manager.cc:1918] T 863dd112e42f46dd9759a7fb11821ef1 P 2c62ec1636f34095a1fa40b04f80da9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:05:44.718371 13413 log.cc:1196] T 863dd112e42f46dd9759a7fb11821ef1 P 2c62ec1636f34095a1fa40b04f80da9e: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-1/wal/wals/863dd112e42f46dd9759a7fb11821ef1
I20260807 07:05:44.718906 13413 ts_tablet_manager.cc:1939] T 863dd112e42f46dd9759a7fb11821ef1 P 2c62ec1636f34095a1fa40b04f80da9e: Deleting consensus metadata
I20260807 07:05:44.720224 12647 catalog_manager.cc:4841] TS 2c62ec1636f34095a1fa40b04f80da9e (127.21.83.2:46577): tablet 863dd112e42f46dd9759a7fb11821ef1 (table test-table [id=91dbe99a058d4cd3a31a6e1a62d44640]) successfully deleted
I20260807 07:05:44.841975 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 12726
I20260807 07:05:44.873380 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 12955
I20260807 07:05:44.888634 13369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:44.874279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48732 (local address 127.21.83.2:46577)
0807 07:05:44.874489 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0807 07:05:44.874496 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:05:44.874519 (+ 23us) server_negotiation.cc:407] Connection header received
0807 07:05:44.874632 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:05:44.874638 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:05:44.874728 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:05:44.874834 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:05:44.876121 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:44.877333 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:44.878704 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:44.879378 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:44.879505 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:05:44.881988 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:05:44.882003 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:05:44.882023 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:05:44.882062 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:05:44.885943 (+ 3881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:05:44.886571 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:05:44.886584 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:05:44.886587 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:05:44.886756 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:05:44.887094 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:05:44.887100 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:05:44.887103 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:05:44.887525 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:05:44.887878 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0807 07:05:44.888022 (+ 144us) server_negotiation.cc:299] Negotiation successful
0807 07:05:44.888233 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 07:05:44.889400 13146 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:05:45.009359 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 12955
I20260807 07:05:45.033747 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 13157
I20260807 07:05:45.046481 13368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:05:45.034470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51340 (local address 127.21.83.3:40235)
0807 07:05:45.034638 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0807 07:05:45.034642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:05:45.034660 (+ 18us) server_negotiation.cc:407] Connection header received
0807 07:05:45.034791 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:05:45.034795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:05:45.034876 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:05:45.034967 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:05:45.036202 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:45.036972 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:45.038221 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:05:45.038760 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:05:45.038859 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:05:45.041476 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:05:45.041490 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:05:45.041504 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:05:45.041536 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:05:45.044167 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:05:45.044728 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:05:45.044734 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:05:45.044737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:05:45.044870 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:05:45.045318 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:05:45.045321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:05:45.045323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:05:45.045652 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:05:45.045863 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0807 07:05:45.046082 (+ 219us) server_negotiation.cc:299] Negotiation successful
0807 07:05:45.046245 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 07:05:45.047075 13339 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:05:45.135699 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 13157
I20260807 07:05:45.158397 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 12618
I20260807 07:05:45.159487 12717 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:05:45.272535 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 12618
I20260807 07:06:05.298849 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:39001
--webserver_interface=127.21.83.62
--webserver_port=46785
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:38161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:39001
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:06:05.653012 13418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:06:05.653476 13418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:06:05.653649 13418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:06:05.664100 13418 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260807 07:06:05.664192 13418 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260807 07:06:05.664237 13418 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260807 07:06:05.664287 13418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:06:05.664341 13418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:06:05.664410 13418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:06:05.664456 13418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:06:05.664508 13418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:06:05.674400 13418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:38161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:39001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:39001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.83.62
--webserver_port=46785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:06:05.675565 13418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:06:05.676788 13418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:06:05.688546 13428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:05.688460 13429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:05.688521 13432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:06:05.688942 13418 server_base.cc:1034] running on GCE node
I20260807 07:06:05.689952 13418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:06:05.691489 13418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:06:05.692737 13418 hybrid_clock.cc:648] HybridClock initialized: now 1786086365692664 us; error 44 us; skew 500 ppm
I20260807 07:06:05.693090 13418 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:06:05 dist-test-slave-ncp9 krb5kdc[11828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086365, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:06:05.696277 13418 init.cc:377] Logged in from keytab as kudu/127.21.83.62@KRBTEST.COM (short username kudu)
I20260807 07:06:05.697813 13418 webserver.cc:460] Webserver started at http://127.21.83.62:46785/ using document root <none> and password file <none>
I20260807 07:06:05.698493 13418 fs_manager.cc:359] Metadata directory not provided
I20260807 07:06:05.698596 13418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:06:05.704556 13418 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260807 07:06:05.707666 13440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:05.709079 13418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260807 07:06:05.709237 13418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "dfc9b269350e44038e90a41e5b96c1f5"
format_stamp: "Formatted at 2026-08-07 07:05:19 on dist-test-slave-ncp9"
server_key: "229338d6ef901aca7a97ec9540716f87"
server_key_iv: "66fbb33c791cd5224cd12e9d5f734f93"
server_key_version: "encryptionkey@0"
I20260807 07:06:05.710223 13418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:06:05.721650 13418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:06:05.725883 13418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:06:05.726136 13418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:06:05.746580 13418 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.62:39001
I20260807 07:06:05.746613 13518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.62:39001 every 8 connection(s)
I20260807 07:06:05.748075 13418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:06:05.752457 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 13418
I20260807 07:06:05.753377 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.1:32771
--local_ip_for_outbound_sockets=127.21.83.1
--tserver_master_addrs=127.21.83.62:39001
--webserver_port=34403
--webserver_interface=127.21.83.1
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab
--principal=kudu/127.21.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:38161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 07:06:05.754912 13519 sys_catalog.cc:263] Verifying existing consensus state
I20260807 07:06:05.758061 13519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5: Bootstrap starting.
I20260807 07:06:05.785537 13519 log.cc:824] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5: Log is configured to *not* fsync() on all Append() calls
I20260807 07:06:05.807648 13519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 07:06:05.808349 13519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5: Bootstrap complete.
I20260807 07:06:05.821938 13519 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc9b269350e44038e90a41e5b96c1f5" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39001 } }
I20260807 07:06:05.823212 13519 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfc9b269350e44038e90a41e5b96c1f5, State: Initialized, Role: FOLLOWER
I20260807 07:06:05.823944 13519 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc9b269350e44038e90a41e5b96c1f5" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39001 } }
I20260807 07:06:05.824254 13519 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:06:05.824411 13519 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:06:05.824617 13519 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5 [term 2 FOLLOWER]: Advancing to term 3
I20260807 07:06:05.829834 13519 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc9b269350e44038e90a41e5b96c1f5" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39001 } }
I20260807 07:06:05.830288 13519 leader_election.cc:302] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfc9b269350e44038e90a41e5b96c1f5; no voters:
I20260807 07:06:05.830904 13519 leader_election.cc:288] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260807 07:06:05.831048 13524 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 07:06:05.832273 13524 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5 [term 3 LEADER]: Becoming Leader. State: Replica: dfc9b269350e44038e90a41e5b96c1f5, State: Running, Role: LEADER
I20260807 07:06:05.832840 13524 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc9b269350e44038e90a41e5b96c1f5" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39001 } }
I20260807 07:06:05.833138 13519 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:06:05.835304 13526 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfc9b269350e44038e90a41e5b96c1f5. Latest consensus state: current_term: 3 leader_uuid: "dfc9b269350e44038e90a41e5b96c1f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc9b269350e44038e90a41e5b96c1f5" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39001 } } }
I20260807 07:06:05.835729 13526 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5 [sys.catalog]: This master's current role is: LEADER
I20260807 07:06:05.836315 13525 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "dfc9b269350e44038e90a41e5b96c1f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc9b269350e44038e90a41e5b96c1f5" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39001 } } }
I20260807 07:06:05.836619 13525 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5 [sys.catalog]: This master's current role is: LEADER
I20260807 07:06:05.846671 13532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:06:05.856055 13532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:06:05.857657 13532 catalog_manager.cc:1259] Loaded cluster ID: 0f5b480469e24dcfbc8546b45d643a8d
I20260807 07:06:05.857816 13532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:06:05.861918 13532 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:06:05.863797 13532 catalog_manager.cc:5879] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5: Loaded TSK: 0
I20260807 07:06:05.864023 13532 catalog_manager.cc:5879] T 00000000000000000000000000000000 P dfc9b269350e44038e90a41e5b96c1f5: Loaded TSK: 1
F20260807 07:06:05.865231 13532 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f00f67099ad google::LogMessage::Fail() at ??:0
@ 0x7f00f670c88a google::LogMessage::SendToLog() at ??:0
@ 0x7f00f67094de google::LogMessage::Flush() at ??:0
@ 0x7f00f670aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20260807 07:06:05.877214 13548 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f00fa46d7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f00fa45c73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f010c1381c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f010c137783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f010c1c32a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f010c1c30c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f010c20d5e9 std::function<>::operator()() at ??:0
@ 0x7f010c13bcdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f010c139c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f010c1c1610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f010c1c1439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f00f9355345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f00f935891c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f00f9358799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f00f9333eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f00fc60d6db start_thread at ??:0
@ 0x7f00f55a361f clone at ??:0
W20260807 07:06:06.183430 13522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:06:06.183723 13522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:06:06.183822 13522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:06:06.191156 13522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:06:06.191290 13522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:06:06.191412 13522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.1
I20260807 07:06:06.201380 13522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:38161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab
--principal=kudu/127.21.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.1:32771
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.83.1
--webserver_port=34403
--tserver_master_addrs=127.21.83.62:39001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.1
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:06:06.202746 13522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:06:06.204020 13522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:06:06.217296 13581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:06.217299 13583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:06.217239 13585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:06:06.218097 13522 server_base.cc:1034] running on GCE node
I20260807 07:06:06.218783 13522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:06:06.219892 13522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:06:06.221132 13522 hybrid_clock.cc:648] HybridClock initialized: now 1786086366221071 us; error 53 us; skew 500 ppm
I20260807 07:06:06.221530 13522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:06:06 dist-test-slave-ncp9 krb5kdc[11828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086366, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:06:06.225112 13522 init.cc:377] Logged in from keytab as kudu/127.21.83.1@KRBTEST.COM (short username kudu)
I20260807 07:06:06.226838 13522 webserver.cc:460] Webserver started at http://127.21.83.1:34403/ using document root <none> and password file <none>
I20260807 07:06:06.227584 13522 fs_manager.cc:359] Metadata directory not provided
I20260807 07:06:06.227720 13522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:06:06.233708 13522 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260807 07:06:06.236737 13597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:06.238173 13522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 07:06:06.238339 13522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-0/wal
uuid: "faefba3fc057402dae24f9e78dfdf048"
format_stamp: "Formatted at 2026-08-07 07:05:20 on dist-test-slave-ncp9"
server_key: "70ce1b31c472ebdce137a2a6cf576add"
server_key_iv: "c907f32db2236ebaf31cab5bcc6d84c3"
server_key_version: "encryptionkey@0"
I20260807 07:06:06.239346 13522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:06:06.249920 13522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:06:06.253800 13522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:06:06.254047 13522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:06:06.254952 13522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:06:06.256181 13522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:06:06.256274 13522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:06.256376 13522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:06:06.256445 13522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:06.288187 13522 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.1:32771
I20260807 07:06:06.288268 13755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.1:32771 every 8 connection(s)
I20260807 07:06:06.289659 13522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
I20260807 07:06:06.294027 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 13522
I20260807 07:06:06.294672 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:06.292641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.1:52901)
0807 07:06:06.293462 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:06.293761 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":350,"threads_started":1}
I20260807 07:06:06.294744 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.2:46577
--local_ip_for_outbound_sockets=127.21.83.2
--tserver_master_addrs=127.21.83.62:39001
--webserver_port=45977
--webserver_interface=127.21.83.2
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab
--principal=kudu/127.21.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:38161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:06:06.295522 13756 heartbeater.cc:644] Failed to heartbeat to 127.21.83.62:39001 (0 consecutive failures): Network error: Failed to ping master at 127.21.83.62:39001: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
I20260807 07:06:06.296500 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:06.296120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.1:34487)
0807 07:06:06.296311 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:06.296382 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260807 07:06:06.297556 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:06.297180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.1:53951)
0807 07:06:06.297339 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:06.297421 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260807 07:06:06.297818 13756 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 07:06:06.642637 13761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:06:06.642901 13761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:06:06.643025 13761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:06:06.650192 13761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:06:06.650336 13761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:06:06.650487 13761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.2
I20260807 07:06:06.659936 13761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:38161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab
--principal=kudu/127.21.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.2:46577
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.83.2
--webserver_port=45977
--tserver_master_addrs=127.21.83.62:39001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.2
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:06:06.661173 13761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:06:06.662521 13761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:06:06.674831 13766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:06.674865 13767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:06.674831 13771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:06:06.675937 13761 server_base.cc:1034] running on GCE node
I20260807 07:06:06.676498 13761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:06:06.677414 13761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:06:06.678603 13761 hybrid_clock.cc:648] HybridClock initialized: now 1786086366678561 us; error 35 us; skew 500 ppm
I20260807 07:06:06.678897 13761 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:06:06 dist-test-slave-ncp9 krb5kdc[11828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086366, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:06:06.681915 13761 init.cc:377] Logged in from keytab as kudu/127.21.83.2@KRBTEST.COM (short username kudu)
I20260807 07:06:06.683315 13761 webserver.cc:460] Webserver started at http://127.21.83.2:45977/ using document root <none> and password file <none>
I20260807 07:06:06.683971 13761 fs_manager.cc:359] Metadata directory not provided
I20260807 07:06:06.684082 13761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:06:06.689340 13761 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260807 07:06:06.691905 13786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:06.693190 13761 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 07:06:06.693344 13761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-1/wal
uuid: "2c62ec1636f34095a1fa40b04f80da9e"
format_stamp: "Formatted at 2026-08-07 07:05:20 on dist-test-slave-ncp9"
server_key: "d50e3a0215607a0e4ffa9fdf98d2671b"
server_key_iv: "fd4e230fae2fb90a9775d32ffa2ccdb9"
server_key_version: "encryptionkey@0"
I20260807 07:06:06.694319 13761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:06:06.708776 13761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:06:06.712635 13761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:06:06.712877 13761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:06:06.713866 13761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:06:06.715188 13761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:06:06.715276 13761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:06.715374 13761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:06:06.715448 13761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:06.747682 13761 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.2:46577
I20260807 07:06:06.747790 13950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.2:46577 every 8 connection(s)
I20260807 07:06:06.749405 13761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
I20260807 07:06:06.754841 13954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:06.752838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.2:38955)
0807 07:06:06.753708 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:06.753932 (+ 224us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":327,"threads_started":1}
W20260807 07:06:06.755587 13951 heartbeater.cc:644] Failed to heartbeat to 127.21.83.62:39001 (0 consecutive failures): Network error: Failed to ping master at 127.21.83.62:39001: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
I20260807 07:06:06.756548 13954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:06.756199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.2:55325)
0807 07:06:06.756356 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:06.756430 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260807 07:06:06.757560 13954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:06.757193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.2:44611)
0807 07:06:06.757326 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:06.757428 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260807 07:06:06.757839 13951 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260807 07:06:06.758188 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 13761
I20260807 07:06:06.758801 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.3:40235
--local_ip_for_outbound_sockets=127.21.83.3
--tserver_master_addrs=127.21.83.62:39001
--webserver_port=41749
--webserver_interface=127.21.83.3
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab
--principal=kudu/127.21.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:38161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:06:07.097119 13956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:06:07.097409 13956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:06:07.097591 13956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:06:07.105834 13956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:06:07.105973 13956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:06:07.106130 13956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.3
I20260807 07:06:07.116575 13956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:38161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab
--principal=kudu/127.21.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.3:40235
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.83.3
--webserver_port=41749
--tserver_master_addrs=127.21.83.62:39001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.3
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:06:07.117930 13956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:06:07.119269 13956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:06:07.132335 13967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:07.132488 13974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:07.132947 13971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:06:07.133702 13956 server_base.cc:1034] running on GCE node
I20260807 07:06:07.134328 13956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:06:07.135314 13956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:06:07.136529 13956 hybrid_clock.cc:648] HybridClock initialized: now 1786086367136474 us; error 48 us; skew 500 ppm
I20260807 07:06:07.136850 13956 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:06:07 dist-test-slave-ncp9 krb5kdc[11828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086367, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:06:07.139885 13956 init.cc:377] Logged in from keytab as kudu/127.21.83.3@KRBTEST.COM (short username kudu)
I20260807 07:06:07.141384 13956 webserver.cc:460] Webserver started at http://127.21.83.3:41749/ using document root <none> and password file <none>
I20260807 07:06:07.142037 13956 fs_manager.cc:359] Metadata directory not provided
I20260807 07:06:07.142146 13956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:06:07.147635 13956 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20260807 07:06:07.150353 13984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:07.151774 13956 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 07:06:07.151954 13956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-2/wal
uuid: "0c43f4e5f9414fa6b99d9c0a7c0dd30c"
format_stamp: "Formatted at 2026-08-07 07:05:21 on dist-test-slave-ncp9"
server_key: "044ff0c34bf96ac04c77383f7f3a477a"
server_key_iv: "f7f629e11f8723ad1e524c108555ca30"
server_key_version: "encryptionkey@0"
I20260807 07:06:07.153283 13956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:06:07.179354 13956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:06:07.183213 13956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:06:07.183449 13956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:06:07.184332 13956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:06:07.185571 13956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:06:07.185658 13956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:07.185758 13956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:06:07.185822 13956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:07.216163 13956 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.3:40235
I20260807 07:06:07.216255 14164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.3:40235 every 8 connection(s)
I20260807 07:06:07.217530 13956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
I20260807 07:06:07.221873 14172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:07.220150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.3:57787)
0807 07:06:07.220828 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:07.221084 (+ 256us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":259,"threads_started":1}
W20260807 07:06:07.222440 14166 heartbeater.cc:644] Failed to heartbeat to 127.21.83.62:39001 (0 consecutive failures): Network error: Failed to ping master at 127.21.83.62:39001: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
I20260807 07:06:07.223124 14172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:07.222852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.3:58933)
0807 07:06:07.222964 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:07.223027 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260807 07:06:07.223817 14172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:07.223540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.3:39819)
0807 07:06:07.223648 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:07.223715 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260807 07:06:07.224011 14166 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260807 07:06:07.226682 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 13956
I20260807 07:06:07.299168 14174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:07.298338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.1:49239)
0807 07:06:07.298806 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:07.298887 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":229,"threads_started":1}
I20260807 07:06:07.759182 14176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:07.758303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.2:37613)
0807 07:06:07.758781 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:07.758906 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":225,"threads_started":1}
I20260807 07:06:08.225471 14177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:08.224579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.3:42145)
0807 07:06:08.225123 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:08.225221 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":252,"threads_started":1}
I20260807 07:06:08.300815 14180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:08.300055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.1:34737)
0807 07:06:08.300553 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:08.300636 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":248,"threads_started":1}
I20260807 07:06:08.760679 14181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:08.760003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.2:60937)
0807 07:06:08.760449 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:08.760526 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":210,"threads_started":1}
I20260807 07:06:09.226992 14182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:09.226303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.3:46417)
0807 07:06:09.226753 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:09.226834 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":231,"threads_started":1}
I20260807 07:06:09.302369 14185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:09.301617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.1:58633)
0807 07:06:09.302089 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:09.302176 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":231,"threads_started":1}
I20260807 07:06:09.762176 14186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:09.761431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.2:38813)
0807 07:06:09.761874 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:09.761962 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":230,"threads_started":1}
I20260807 07:06:10.228554 14188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:10.227867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.3:42923)
0807 07:06:10.228299 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:10.228377 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":233,"threads_started":1}
I20260807 07:06:10.303869 14189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:10.303175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.1:34407)
0807 07:06:10.303599 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:10.303678 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":201,"threads_started":1}
I20260807 07:06:10.763939 14190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:10.763129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.2:49585)
0807 07:06:10.763674 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:10.763754 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":356,"threads_started":1}
I20260807 07:06:11.230386 14191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:11.229584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.3:40211)
0807 07:06:11.230097 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:11.230187 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":274,"threads_started":1}
I20260807 07:06:11.305572 14192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:11.304813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.1:34657)
0807 07:06:11.305313 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:11.305395 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":287,"threads_started":1}
I20260807 07:06:11.765460 14197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:11.764750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.2:43537)
0807 07:06:11.765224 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:11.765301 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":268,"threads_started":1}
I20260807 07:06:12.231935 14198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:12.231232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.3:41277)
0807 07:06:12.231685 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:12.231765 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":250,"threads_started":1}
I20260807 07:06:12.307083 14199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:12.306378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.1:53533)
0807 07:06:12.306795 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:12.306882 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":217,"threads_started":1}
I20260807 07:06:12.766988 14200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:12.766275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.2:43949)
0807 07:06:12.766738 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:12.766829 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":238,"threads_started":1}
I20260807 07:06:13.233611 14201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:13.232842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.3:45893)
0807 07:06:13.233348 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:13.233440 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":283,"threads_started":1}
I20260807 07:06:13.308764 14203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:13.307892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.1:47441)
0807 07:06:13.308417 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:13.308500 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":248,"threads_started":1}
I20260807 07:06:13.769017 14206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:13.768001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.2:47457)
0807 07:06:13.768618 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:13.768729 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":266,"threads_started":1}
I20260807 07:06:14.235514 14208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:14.234613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.3:56155)
0807 07:06:14.235179 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:14.235275 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":256,"threads_started":1}
I20260807 07:06:14.310571 14209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:14.309737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.1:34649)
0807 07:06:14.310252 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:14.310342 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":267,"threads_started":1}
I20260807 07:06:14.770584 14210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:14.769875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.2:38969)
0807 07:06:14.770327 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:14.770406 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":206,"threads_started":1}
I20260807 07:06:15.237263 14212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:15.236503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.3:47321)
0807 07:06:15.236993 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:15.237086 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":232,"threads_started":1}
I20260807 07:06:15.312341 14213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:15.311595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.1:47219)
0807 07:06:15.312080 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:15.312163 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":244,"threads_started":1}
I20260807 07:06:15.772130 14215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:15.771363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.2:53179)
0807 07:06:15.771847 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:15.771929 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":246,"threads_started":1}
I20260807 07:06:16.238929 14217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:16.238168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.3:54261)
0807 07:06:16.238644 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:16.238741 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":225,"threads_started":1}
I20260807 07:06:16.314039 14218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:16.313277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.1:53483)
0807 07:06:16.313798 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:16.313878 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":279,"threads_started":1}
I20260807 07:06:16.773744 14220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:16.773022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.2:53329)
0807 07:06:16.773486 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:16.773568 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":251,"threads_started":1}
I20260807 07:06:17.240798 14221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:17.239985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.3:57173)
0807 07:06:17.240484 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:17.240608 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":265,"threads_started":1}
I20260807 07:06:17.315912 14222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:17.315030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.1:52373)
0807 07:06:17.315548 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:17.315639 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":268,"threads_started":1}
I20260807 07:06:17.775369 14223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:17.774616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.2:47869)
0807 07:06:17.775093 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:17.775176 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":232,"threads_started":1}
I20260807 07:06:18.242403 14225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:18.241655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.3:43345)
0807 07:06:18.242097 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:18.242210 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":225,"threads_started":1}
I20260807 07:06:18.317698 14227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:18.316841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.1:32907)
0807 07:06:18.317427 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:18.317530 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":286,"threads_started":1}
I20260807 07:06:18.777069 14230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:18.776271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.2:59063)
0807 07:06:18.776767 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:18.776855 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":252,"threads_started":1}
I20260807 07:06:19.244019 14232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:19.243275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.3:53075)
0807 07:06:19.243745 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:19.243828 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":251,"threads_started":1}
I20260807 07:06:19.319511 14233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:19.318656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.1:37529)
0807 07:06:19.319208 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:19.319332 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":280,"threads_started":1}
I20260807 07:06:19.778750 14234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:19.777971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.2:46953)
0807 07:06:19.778446 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:19.778562 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":245,"threads_started":1}
I20260807 07:06:20.245669 14235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:20.244850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.3:54201)
0807 07:06:20.245376 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:20.245474 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":269,"threads_started":1}
I20260807 07:06:20.321121 14236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:20.320373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.1:52185)
0807 07:06:20.320812 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:20.320920 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":216,"threads_started":1}
I20260807 07:06:20.780553 14237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:20.779803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.2:35057)
0807 07:06:20.780274 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:20.780367 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":231,"threads_started":1}
I20260807 07:06:21.247298 14238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:21.246492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.3:36977)
0807 07:06:21.246997 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:21.247089 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":256,"threads_started":1}
I20260807 07:06:21.323153 14239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:21.322352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.1:54309)
0807 07:06:21.322847 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:21.322963 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":257,"threads_started":1}
I20260807 07:06:21.782716 14241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:21.781807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.2:55277)
0807 07:06:21.782318 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:21.782431 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":249,"threads_started":1}
I20260807 07:06:22.231431 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 13522
I20260807 07:06:22.249507 14246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:22.248594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.3:42881)
0807 07:06:22.249183 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:22.249316 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":323,"threads_started":1}
I20260807 07:06:22.258756 14242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:22.232811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45400 (local address 127.21.83.1:32771)
0807 07:06:22.233556 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:22.233577 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:22.233634 (+ 57us) server_negotiation.cc:407] Connection header received
0807 07:06:22.235475 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:22.235487 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:22.235966 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:22.236301 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:22.238012 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:22.239342 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:22.240701 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:22.241478 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:22.241648 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:22.252335 (+ 10687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:22.252346 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:22.252365 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:06:22.252417 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:22.255427 (+ 3010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:22.256217 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:22.256223 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:22.256230 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:22.256417 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:22.256725 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:22.256727 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:22.256729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:22.257384 (+ 655us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:22.257633 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:22.258024 (+ 391us) server_negotiation.cc:299] Negotiation successful
0807 07:06:22.258459 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":379,"threads_started":1}
I20260807 07:06:22.260232 13750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:06:22.346838 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 13522
I20260807 07:06:22.346896 14250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:22.345685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39001 (local address 127.21.83.1:52339)
0807 07:06:22.346261 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:22.346519 (+ 258us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.62:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":275,"threads_started":1}
I20260807 07:06:22.374850 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 13761
I20260807 07:06:22.392176 14252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:22.375757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56464 (local address 127.21.83.2:46577)
0807 07:06:22.376345 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:22.376359 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:22.376395 (+ 36us) server_negotiation.cc:407] Connection header received
0807 07:06:22.378060 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:22.378071 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:22.378488 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:22.378789 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:22.380090 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:22.381537 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:22.382892 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:22.383605 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:22.383739 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:22.386283 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:22.386290 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:22.386305 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:06:22.386347 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:22.389309 (+ 2962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:22.389941 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:22.389947 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:22.389953 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:22.390131 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:22.390404 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:22.390407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:22.390409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:22.390921 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:22.391156 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:22.391473 (+ 317us) server_negotiation.cc:299] Negotiation successful
0807 07:06:22.391883 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":273,"threads_started":1}
I20260807 07:06:22.393760 13946 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:06:22.477346 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 13761
I20260807 07:06:22.501516 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 13956
I20260807 07:06:22.518335 14254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:22.502434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34784 (local address 127.21.83.3:40235)
0807 07:06:22.503055 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:22.503069 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:22.503109 (+ 40us) server_negotiation.cc:407] Connection header received
0807 07:06:22.504756 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:22.504765 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:22.505224 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:22.505500 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:22.506800 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:22.507961 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:22.509360 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:22.509982 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:22.510114 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:22.512695 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:22.512702 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:22.512715 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:06:22.512753 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:22.515515 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:22.516114 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:22.516120 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:22.516125 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:22.516283 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:22.516576 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:22.516578 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:22.516580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:22.517111 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:22.517339 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:22.517641 (+ 302us) server_negotiation.cc:299] Negotiation successful
0807 07:06:22.518021 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":279,"threads_started":1}
I20260807 07:06:22.519621 14160 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:06:22.601931 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 13956
2026-08-07T07:06:22Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64764 ms)
[----------] 30 tests from SecurityITest (304489 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 07:06:22 dist-test-slave-ncp9 krb5kdc[14265](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 07:06:22 dist-test-slave-ncp9 krb5kdc[14265](info): set up 2 sockets
Aug 07 07:06:22 dist-test-slave-ncp9 krb5kdc[14265](info): commencing operation
krb5kdc: starting...
W20260807 07:06:23.848021 21836 mini_kdc.cc:121] Time spent starting KDC: real 1.201s user 0.000s sys 0.056s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Aug 07 07:06:24 dist-test-slave-ncp9 krb5kdc[14265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T07:06:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:06:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
I20260807 07:06:24.208047 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:43873
--webserver_interface=127.21.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:39127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:43873
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:06:24.568482 14283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:06:24.568773 14283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:06:24.568920 14283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:06:24.576231 14283 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:06:24.576341 14283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:06:24.576400 14283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:06:24.576481 14283 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:06:24.576546 14283 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:06:24.586540 14283 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:39127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:43873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:43873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:06:24.587848 14283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:06:24.589201 14283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:06:24.600870 14291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:24.600811 14293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:24.600872 14289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:06:24.601753 14283 server_base.cc:1034] running on GCE node
I20260807 07:06:24.602644 14283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:06:24.604126 14283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:06:24.605334 14283 hybrid_clock.cc:648] HybridClock initialized: now 1786086384605285 us; error 33 us; skew 500 ppm
I20260807 07:06:24.605705 14283 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:06:24 dist-test-slave-ncp9 krb5kdc[14265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086384, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:06:24.608722 14283 init.cc:377] Logged in from keytab as kudu/127.21.83.62@KRBTEST.COM (short username kudu)
I20260807 07:06:24.610304 14283 webserver.cc:460] Webserver started at http://127.21.83.62:39317/ using document root <none> and password file <none>
I20260807 07:06:24.610981 14283 fs_manager.cc:359] Metadata directory not provided
I20260807 07:06:24.611104 14283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:06:24.611398 14283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:06:24.614446 14283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "70c2622b0ea94f86a72dddaee910157c"
format_stamp: "Formatted at 2026-08-07 07:06:24 on dist-test-slave-ncp9"
server_key: "84523065821ac6ecbb06cb71bd4d7538"
server_key_iv: "b72937007b4ea76cf17121986d10ff6d"
server_key_version: "encryptionkey@0"
I20260807 07:06:24.615208 14283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "70c2622b0ea94f86a72dddaee910157c"
format_stamp: "Formatted at 2026-08-07 07:06:24 on dist-test-slave-ncp9"
server_key: "84523065821ac6ecbb06cb71bd4d7538"
server_key_iv: "b72937007b4ea76cf17121986d10ff6d"
server_key_version: "encryptionkey@0"
I20260807 07:06:24.620432 14283 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260807 07:06:24.624228 14304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:24.625682 14283 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 07:06:24.625871 14283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "70c2622b0ea94f86a72dddaee910157c"
format_stamp: "Formatted at 2026-08-07 07:06:24 on dist-test-slave-ncp9"
server_key: "84523065821ac6ecbb06cb71bd4d7538"
server_key_iv: "b72937007b4ea76cf17121986d10ff6d"
server_key_version: "encryptionkey@0"
I20260807 07:06:24.626080 14283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:06:24.636566 14283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:06:24.640301 14283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:06:24.640581 14283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:06:24.659021 14283 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.62:43873
I20260807 07:06:24.659080 14385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.62:43873 every 8 connection(s)
I20260807 07:06:24.660523 14283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:06:24.663522 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 14283
I20260807 07:06:24.663784 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:06:24.664711 21836 external_mini_cluster.cc:1428] Setting key ae781a4fa830ecc6912ce15b97675f12
I20260807 07:06:24.665364 14388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:06:24.681411 14388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70c2622b0ea94f86a72dddaee910157c: Bootstrap starting.
Aug 07 07:06:24 dist-test-slave-ncp9 krb5kdc[14265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:06:24.684881 14388 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70c2622b0ea94f86a72dddaee910157c: Neither blocks nor log segments found. Creating new log.
I20260807 07:06:24.686067 14388 log.cc:824] T 00000000000000000000000000000000 P 70c2622b0ea94f86a72dddaee910157c: Log is configured to *not* fsync() on all Append() calls
I20260807 07:06:24.689232 14388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70c2622b0ea94f86a72dddaee910157c: No bootstrap required, opened a new log
I20260807 07:06:24.690634 14394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:24.667171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56892 (local address 127.21.83.62:43873)
0807 07:06:24.668034 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:24.668049 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:24.668096 (+ 47us) server_negotiation.cc:407] Connection header received
0807 07:06:24.670496 (+ 2400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:24.670527 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:24.671031 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:24.671447 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:24.673727 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:24.675116 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:24.677105 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:24.677927 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:24.678088 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:24.683210 (+ 5122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:24.683219 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:24.683237 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:06:24.683285 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:24.686421 (+ 3136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:24.687220 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:24.687228 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:24.687234 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:24.687421 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:24.687837 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:24.687841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:24.687843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:24.688405 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:24.688701 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:24.689209 (+ 508us) server_negotiation.cc:299] Negotiation successful
0807 07:06:24.689648 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":384,"threads_started":1}
I20260807 07:06:24.697041 14388 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70c2622b0ea94f86a72dddaee910157c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70c2622b0ea94f86a72dddaee910157c" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 43873 } }
I20260807 07:06:24.697391 14388 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70c2622b0ea94f86a72dddaee910157c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:06:24.697492 14388 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70c2622b0ea94f86a72dddaee910157c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70c2622b0ea94f86a72dddaee910157c, State: Initialized, Role: FOLLOWER
I20260807 07:06:24.698016 14388 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70c2622b0ea94f86a72dddaee910157c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70c2622b0ea94f86a72dddaee910157c" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 43873 } }
I20260807 07:06:24.698251 14388 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70c2622b0ea94f86a72dddaee910157c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:06:24.698375 14388 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70c2622b0ea94f86a72dddaee910157c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:06:24.698544 14388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70c2622b0ea94f86a72dddaee910157c [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:06:24.701061 14388 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70c2622b0ea94f86a72dddaee910157c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70c2622b0ea94f86a72dddaee910157c" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 43873 } }
I20260807 07:06:24.701557 14388 leader_election.cc:302] T 00000000000000000000000000000000 P 70c2622b0ea94f86a72dddaee910157c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70c2622b0ea94f86a72dddaee910157c; no voters:
I20260807 07:06:24.702104 14388 leader_election.cc:288] T 00000000000000000000000000000000 P 70c2622b0ea94f86a72dddaee910157c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 07:06:24.702252 14396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70c2622b0ea94f86a72dddaee910157c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:06:24.702901 14396 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70c2622b0ea94f86a72dddaee910157c [term 1 LEADER]: Becoming Leader. State: Replica: 70c2622b0ea94f86a72dddaee910157c, State: Running, Role: LEADER
I20260807 07:06:24.703433 14396 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70c2622b0ea94f86a72dddaee910157c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70c2622b0ea94f86a72dddaee910157c" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 43873 } }
I20260807 07:06:24.704119 14388 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70c2622b0ea94f86a72dddaee910157c [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:06:24.706120 14397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70c2622b0ea94f86a72dddaee910157c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70c2622b0ea94f86a72dddaee910157c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70c2622b0ea94f86a72dddaee910157c" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 43873 } } }
I20260807 07:06:24.706460 14397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70c2622b0ea94f86a72dddaee910157c [sys.catalog]: This master's current role is: LEADER
I20260807 07:06:24.707324 14401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:06:24.707188 14398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70c2622b0ea94f86a72dddaee910157c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70c2622b0ea94f86a72dddaee910157c. Latest consensus state: current_term: 1 leader_uuid: "70c2622b0ea94f86a72dddaee910157c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70c2622b0ea94f86a72dddaee910157c" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 43873 } } }
I20260807 07:06:24.708452 14398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70c2622b0ea94f86a72dddaee910157c [sys.catalog]: This master's current role is: LEADER
I20260807 07:06:24.713039 14401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:06:24.723419 14401 catalog_manager.cc:1347] Generated new cluster ID: a6c1fd494937424cb6947b8f9ad2bee2
I20260807 07:06:24.723558 14401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:06:24.732962 14401 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:06:24.734373 14401 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:06:24.751587 14401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70c2622b0ea94f86a72dddaee910157c: Generated new TSK 0
I20260807 07:06:24.752506 14401 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.83.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab.
Entry for principal kudu/127.21.83.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab.
I20260807 07:06:24.845898 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.1:0
--local_ip_for_outbound_sockets=127.21.83.1
--webserver_interface=127.21.83.1
--webserver_port=0
--tserver_master_addrs=127.21.83.62:43873
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab
--principal=kudu/127.21.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:39127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:06:25.221231 14423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:06:25.221517 14423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:06:25.221642 14423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:06:25.229144 14423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:06:25.229313 14423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:06:25.229511 14423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.1
I20260807 07:06:25.239876 14423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:39127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab
--principal=kudu/127.21.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.83.1
--webserver_port=0
--tserver_master_addrs=127.21.83.62:43873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.1
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:06:25.241339 14423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:06:25.242671 14423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:06:25.255201 14430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:25.255141 14431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:25.255168 14433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:06:25.256220 14423 server_base.cc:1034] running on GCE node
I20260807 07:06:25.256769 14423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:06:25.257643 14423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:06:25.258852 14423 hybrid_clock.cc:648] HybridClock initialized: now 1786086385258797 us; error 41 us; skew 500 ppm
I20260807 07:06:25.259136 14423 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:06:25 dist-test-slave-ncp9 krb5kdc[14265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086385, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:06:25.262449 14423 init.cc:377] Logged in from keytab as kudu/127.21.83.1@KRBTEST.COM (short username kudu)
I20260807 07:06:25.264106 14423 webserver.cc:460] Webserver started at http://127.21.83.1:39655/ using document root <none> and password file <none>
I20260807 07:06:25.264956 14423 fs_manager.cc:359] Metadata directory not provided
I20260807 07:06:25.265102 14423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:06:25.265445 14423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:06:25.268594 14423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-0/data/instance:
uuid: "0c4a23cbb00947848c8c0d5ed8b82236"
format_stamp: "Formatted at 2026-08-07 07:06:25 on dist-test-slave-ncp9"
server_key: "85e6d418b47518b03e64bc41d7114b4b"
server_key_iv: "e706b03cf6e7e2c7fe65b67ad1c58ea8"
server_key_version: "encryptionkey@0"
I20260807 07:06:25.269488 14423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-0/wal/instance:
uuid: "0c4a23cbb00947848c8c0d5ed8b82236"
format_stamp: "Formatted at 2026-08-07 07:06:25 on dist-test-slave-ncp9"
server_key: "85e6d418b47518b03e64bc41d7114b4b"
server_key_iv: "e706b03cf6e7e2c7fe65b67ad1c58ea8"
server_key_version: "encryptionkey@0"
I20260807 07:06:25.274941 14423 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 07:06:25.278517 14443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:25.279835 14423 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260807 07:06:25.280006 14423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-0/wal
uuid: "0c4a23cbb00947848c8c0d5ed8b82236"
format_stamp: "Formatted at 2026-08-07 07:06:25 on dist-test-slave-ncp9"
server_key: "85e6d418b47518b03e64bc41d7114b4b"
server_key_iv: "e706b03cf6e7e2c7fe65b67ad1c58ea8"
server_key_version: "encryptionkey@0"
I20260807 07:06:25.280205 14423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:06:25.294471 14423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:06:25.298658 14423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:06:25.298981 14423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:06:25.300019 14423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:06:25.301503 14423 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:06:25.301597 14423 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:25.301733 14423 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:06:25.301808 14423 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:25.336266 14423 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.1:32999
I20260807 07:06:25.336303 14606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.1:32999 every 8 connection(s)
I20260807 07:06:25.337803 14423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
I20260807 07:06:25.346330 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 14423
I20260807 07:06:25.346529 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-0/wal/instance
I20260807 07:06:25.347224 21836 external_mini_cluster.cc:1428] Setting key afccfe329e5f329a144e966bfd3b6161
Aug 07 07:06:25 dist-test-slave-ncp9 krb5kdc[14265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086385, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.1@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:06:25.359997 14609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:25.340788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:39765 (local address 127.21.83.62:43873)
0807 07:06:25.341295 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:25.341299 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:25.342708 (+ 1409us) server_negotiation.cc:407] Connection header received
0807 07:06:25.345048 (+ 2340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:25.345055 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:25.345159 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:25.345274 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:25.347105 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:25.347885 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:25.349655 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:25.350082 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:25.350217 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:25.353941 (+ 3724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:25.353947 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:25.353950 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:06:25.353990 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:25.356586 (+ 2596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:25.357492 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:25.357497 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:25.357499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:25.357635 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:25.358107 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:25.358111 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:25.358113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:25.358436 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:25.358579 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:25.359370 (+ 791us) server_negotiation.cc:299] Negotiation successful
0807 07:06:25.359628 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":281,"threads_started":1}
I20260807 07:06:25.361097 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:25.341213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:43873 (local address 127.21.83.1:39765)
0807 07:06:25.342506 (+ 1293us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:25.342562 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0807 07:06:25.344659 (+ 2097us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:06:25.345565 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:06:25.345575 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:06:25.346198 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:06:25.346905 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:25.346928 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:25.348082 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:25.348086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:25.349495 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:25.349511 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:25.350346 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:25.350354 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:25.350605 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:25.351302 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:06:25.351327 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:06:25.353657 (+ 2330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:06:25.356779 (+ 3122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:06:25.356787 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:06:25.356803 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:06:25.357349 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:06:25.357754 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:06:25.357759 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:06:25.357763 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:06:25.357981 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:06:25.358574 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:06:25.358583 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:06:25.358967 (+ 384us) client_negotiation.cc:769] Sending connection context
0807 07:06:25.359229 (+ 262us) client_negotiation.cc:240] Negotiation successful
0807 07:06:25.359829 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":816,"thread_start_us":498,"threads_started":1}
I20260807 07:06:25.362846 14607 heartbeater.cc:347] Connected to a master server at 127.21.83.62:43873
I20260807 07:06:25.363162 14607 heartbeater.cc:464] Registering TS with master...
I20260807 07:06:25.363941 14607 heartbeater.cc:511] Master 127.21.83.62:43873 requested a full tablet report, sending...
I20260807 07:06:25.366537 14330 ts_manager.cc:194] Registered new tserver with Master: 0c4a23cbb00947848c8c0d5ed8b82236 (127.21.83.1:32999)
I20260807 07:06:25.368119 14330 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.1@KRBTEST.COM'} at 127.21.83.1:39765
WARNING: no policy specified for kudu/127.21.83.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab.
Entry for principal kudu/127.21.83.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab.
I20260807 07:06:25.424378 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.2:0
--local_ip_for_outbound_sockets=127.21.83.2
--webserver_interface=127.21.83.2
--webserver_port=0
--tserver_master_addrs=127.21.83.62:43873
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab
--principal=kudu/127.21.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:39127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:06:25.794965 14613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:06:25.795274 14613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:06:25.795414 14613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:06:25.802886 14613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:06:25.803020 14613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:06:25.803176 14613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.2
I20260807 07:06:25.813293 14613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:39127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab
--principal=kudu/127.21.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.83.2
--webserver_port=0
--tserver_master_addrs=127.21.83.62:43873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.2
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:06:25.814625 14613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:06:25.815929 14613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:06:25.829016 14619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:25.829030 14620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:25.829005 14624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:06:25.829895 14613 server_base.cc:1034] running on GCE node
I20260807 07:06:25.830557 14613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:06:25.831521 14613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:06:25.832731 14613 hybrid_clock.cc:648] HybridClock initialized: now 1786086385832681 us; error 46 us; skew 500 ppm
I20260807 07:06:25.833076 14613 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:06:25 dist-test-slave-ncp9 krb5kdc[14265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086385, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:06:25.836316 14613 init.cc:377] Logged in from keytab as kudu/127.21.83.2@KRBTEST.COM (short username kudu)
I20260807 07:06:25.837916 14613 webserver.cc:460] Webserver started at http://127.21.83.2:42751/ using document root <none> and password file <none>
I20260807 07:06:25.838604 14613 fs_manager.cc:359] Metadata directory not provided
I20260807 07:06:25.838738 14613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:06:25.839046 14613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:06:25.842096 14613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-1/data/instance:
uuid: "c135d77f3d784aff8092d220514de9d3"
format_stamp: "Formatted at 2026-08-07 07:06:25 on dist-test-slave-ncp9"
server_key: "a5083ba9ed43a443f793bef9dad03476"
server_key_iv: "bf1ccd2de80de5c4377a68508aeaf202"
server_key_version: "encryptionkey@0"
I20260807 07:06:25.842871 14613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-1/wal/instance:
uuid: "c135d77f3d784aff8092d220514de9d3"
format_stamp: "Formatted at 2026-08-07 07:06:25 on dist-test-slave-ncp9"
server_key: "a5083ba9ed43a443f793bef9dad03476"
server_key_iv: "bf1ccd2de80de5c4377a68508aeaf202"
server_key_version: "encryptionkey@0"
I20260807 07:06:25.847975 14613 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260807 07:06:25.851724 14634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:25.853112 14613 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 07:06:25.853267 14613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-1/wal
uuid: "c135d77f3d784aff8092d220514de9d3"
format_stamp: "Formatted at 2026-08-07 07:06:25 on dist-test-slave-ncp9"
server_key: "a5083ba9ed43a443f793bef9dad03476"
server_key_iv: "bf1ccd2de80de5c4377a68508aeaf202"
server_key_version: "encryptionkey@0"
I20260807 07:06:25.853478 14613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:06:25.895273 14613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:06:25.899783 14613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:06:25.900085 14613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:06:25.901118 14613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:06:25.902539 14613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:06:25.902621 14613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:25.902745 14613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:06:25.902798 14613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:25.934509 14613 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.2:37843
I20260807 07:06:25.934571 14791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.2:37843 every 8 connection(s)
I20260807 07:06:25.935814 14613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
I20260807 07:06:25.943061 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 14613
I20260807 07:06:25.943285 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-1/wal/instance
I20260807 07:06:25.943845 21836 external_mini_cluster.cc:1428] Setting key 8f221183c7698e69ddb994d3f0fa1e5c
Aug 07 07:06:25 dist-test-slave-ncp9 krb5kdc[14265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086385, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.2@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:06:25.957827 14795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:25.938514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:39769 (local address 127.21.83.62:43873)
0807 07:06:25.938995 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:25.939001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:25.939979 (+ 978us) server_negotiation.cc:407] Connection header received
0807 07:06:25.941962 (+ 1983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:25.941967 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:25.942060 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:25.942168 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:25.944408 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:25.945375 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:25.947432 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:25.947854 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:25.947950 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:25.952024 (+ 4074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:25.952029 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:25.952033 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:06:25.952068 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:25.954603 (+ 2535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:25.955416 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:25.955420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:25.955422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:25.955547 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:25.955955 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:25.955958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:25.955960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:25.956252 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:25.956369 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:25.957282 (+ 913us) server_negotiation.cc:299] Negotiation successful
0807 07:06:25.957504 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":216,"threads_started":1}
I20260807 07:06:25.959056 14796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:25.939085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:43873 (local address 127.21.83.2:39769)
0807 07:06:25.939804 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:25.939844 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 07:06:25.941646 (+ 1802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:06:25.942448 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:06:25.942457 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:06:25.943348 (+ 891us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:06:25.944195 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:25.944225 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:25.945529 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:25.945533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:25.947294 (+ 1761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:25.947308 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:25.948077 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:25.948083 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:25.948355 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:25.949159 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:06:25.949187 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:06:25.951766 (+ 2579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:06:25.954739 (+ 2973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:06:25.954747 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:06:25.954762 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:06:25.955323 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:06:25.955639 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:06:25.955643 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:06:25.955646 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:06:25.955873 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:06:25.956374 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:06:25.956382 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:06:25.956845 (+ 463us) client_negotiation.cc:769] Sending connection context
0807 07:06:25.957165 (+ 320us) client_negotiation.cc:240] Negotiation successful
0807 07:06:25.957720 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":281,"threads_started":1}
I20260807 07:06:25.960789 14793 heartbeater.cc:347] Connected to a master server at 127.21.83.62:43873
I20260807 07:06:25.961117 14793 heartbeater.cc:464] Registering TS with master...
I20260807 07:06:25.961886 14793 heartbeater.cc:511] Master 127.21.83.62:43873 requested a full tablet report, sending...
I20260807 07:06:25.963603 14330 ts_manager.cc:194] Registered new tserver with Master: c135d77f3d784aff8092d220514de9d3 (127.21.83.2:37843)
I20260807 07:06:25.964401 14330 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.2@KRBTEST.COM'} at 127.21.83.2:39769
WARNING: no policy specified for kudu/127.21.83.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab.
Entry for principal kudu/127.21.83.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab.
I20260807 07:06:26.019541 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.3:0
--local_ip_for_outbound_sockets=127.21.83.3
--webserver_interface=127.21.83.3
--webserver_port=0
--tserver_master_addrs=127.21.83.62:43873
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab
--principal=kudu/127.21.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:39127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 07:06:26.372187 14607 heartbeater.cc:503] Master 127.21.83.62:43873 was elected leader, sending a full tablet report...
W20260807 07:06:26.373962 14800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:06:26.374357 14800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:06:26.374517 14800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:06:26.382171 14800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:06:26.382352 14800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:06:26.382515 14800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.3
I20260807 07:06:26.392984 14800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:39127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab
--principal=kudu/127.21.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.83.3
--webserver_port=0
--tserver_master_addrs=127.21.83.62:43873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.3
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:06:26.394380 14800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:06:26.395699 14800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:06:26.409626 14807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:26.409628 14806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:26.409628 14809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:06:26.410748 14800 server_base.cc:1034] running on GCE node
I20260807 07:06:26.411441 14800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:06:26.412513 14800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:06:26.413744 14800 hybrid_clock.cc:648] HybridClock initialized: now 1786086386413681 us; error 44 us; skew 500 ppm
I20260807 07:06:26.414072 14800 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:06:26 dist-test-slave-ncp9 krb5kdc[14265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086386, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:06:26.417330 14800 init.cc:377] Logged in from keytab as kudu/127.21.83.3@KRBTEST.COM (short username kudu)
I20260807 07:06:26.419271 14800 webserver.cc:460] Webserver started at http://127.21.83.3:41001/ using document root <none> and password file <none>
I20260807 07:06:26.419996 14800 fs_manager.cc:359] Metadata directory not provided
I20260807 07:06:26.420123 14800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:06:26.420447 14800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:06:26.423616 14800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-2/data/instance:
uuid: "64ece599d1ff40698e76fb7377fe4d9a"
format_stamp: "Formatted at 2026-08-07 07:06:26 on dist-test-slave-ncp9"
server_key: "9137377f5bd9dcda25f30b5a4869b269"
server_key_iv: "ed87e226e3006a52ba8fabfd51cb75a0"
server_key_version: "encryptionkey@0"
I20260807 07:06:26.424599 14800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-2/wal/instance:
uuid: "64ece599d1ff40698e76fb7377fe4d9a"
format_stamp: "Formatted at 2026-08-07 07:06:26 on dist-test-slave-ncp9"
server_key: "9137377f5bd9dcda25f30b5a4869b269"
server_key_iv: "ed87e226e3006a52ba8fabfd51cb75a0"
server_key_version: "encryptionkey@0"
I20260807 07:06:26.430465 14800 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 07:06:26.434340 14822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:26.435859 14800 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 07:06:26.436038 14800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-2/wal
uuid: "64ece599d1ff40698e76fb7377fe4d9a"
format_stamp: "Formatted at 2026-08-07 07:06:26 on dist-test-slave-ncp9"
server_key: "9137377f5bd9dcda25f30b5a4869b269"
server_key_iv: "ed87e226e3006a52ba8fabfd51cb75a0"
server_key_version: "encryptionkey@0"
I20260807 07:06:26.436265 14800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:06:26.448736 14800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:06:26.452953 14800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:06:26.453253 14800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:06:26.454260 14800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:06:26.455732 14800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:06:26.455826 14800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:26.455948 14800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:06:26.456008 14800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:26.488549 14800 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.3:38789
I20260807 07:06:26.488595 14989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.3:38789 every 8 connection(s)
I20260807 07:06:26.489925 14800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
I20260807 07:06:26.496805 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 14800
I20260807 07:06:26.497030 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-2/wal/instance
I20260807 07:06:26.497481 21836 external_mini_cluster.cc:1428] Setting key bb1d1d5571f3f6f00fd9217062439843
Aug 07 07:06:26 dist-test-slave-ncp9 krb5kdc[14265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086386, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.3@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:06:26.512743 14994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:26.492551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:40495 (local address 127.21.83.62:43873)
0807 07:06:26.493085 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:26.493089 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:26.493943 (+ 854us) server_negotiation.cc:407] Connection header received
0807 07:06:26.495866 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:26.495872 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:26.495961 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:26.496070 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:26.497865 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:26.498921 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:26.500544 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:26.501017 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:26.501128 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:26.505068 (+ 3940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:26.505075 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:26.505081 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:06:26.505147 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:26.508253 (+ 3106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:26.509308 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:26.509313 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:26.509316 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:26.509467 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:26.510007 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:26.510012 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:26.510014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:26.510318 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:26.510440 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:26.511953 (+ 1513us) server_negotiation.cc:299] Negotiation successful
0807 07:06:26.512223 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":245,"threads_started":1}
I20260807 07:06:26.514180 14995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:26.493001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:43873 (local address 127.21.83.3:40495)
0807 07:06:26.493765 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:26.493809 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 07:06:26.495560 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:06:26.496341 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:06:26.496350 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:06:26.496971 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:06:26.497676 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:26.497697 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:26.499078 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:26.499084 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:26.500387 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:26.500407 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:26.501272 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:26.501280 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:26.501525 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:26.502142 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:06:26.502165 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:06:26.504772 (+ 2607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:06:26.508459 (+ 3687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:06:26.508468 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:06:26.508485 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:06:26.509105 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:06:26.509634 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:06:26.509638 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:06:26.509641 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:06:26.509875 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:06:26.510434 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:06:26.510441 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:06:26.511192 (+ 751us) client_negotiation.cc:769] Sending connection context
0807 07:06:26.511631 (+ 439us) client_negotiation.cc:240] Negotiation successful
0807 07:06:26.512460 (+ 829us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":303,"threads_started":1}
I20260807 07:06:26.516090 14990 heartbeater.cc:347] Connected to a master server at 127.21.83.62:43873
I20260807 07:06:26.516456 14990 heartbeater.cc:464] Registering TS with master...
I20260807 07:06:26.517310 14990 heartbeater.cc:511] Master 127.21.83.62:43873 requested a full tablet report, sending...
I20260807 07:06:26.519197 14330 ts_manager.cc:194] Registered new tserver with Master: 64ece599d1ff40698e76fb7377fe4d9a (127.21.83.3:38789)
I20260807 07:06:26.520269 14330 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.3@KRBTEST.COM'} at 127.21.83.3:40495
I20260807 07:06:26.526167 21836 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 07:06:26.545001 14994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:26.530837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56896 (local address 127.21.83.62:43873)
0807 07:06:26.531066 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:26.531072 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:26.531473 (+ 401us) server_negotiation.cc:407] Connection header received
0807 07:06:26.531644 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:26.531648 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:26.531722 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:26.531808 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:26.533376 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:26.534158 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:26.535556 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:26.535999 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:26.536109 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:26.538694 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:26.538699 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:26.538703 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:06:26.538744 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:26.541613 (+ 2869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:26.542678 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:26.542685 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:26.542688 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:26.543034 (+ 346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:26.543493 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:26.543497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:26.543500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:26.543896 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:26.544100 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:26.544453 (+ 353us) server_negotiation.cc:299] Negotiation successful
0807 07:06:26.544669 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 07:06:26.550002 14330 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56896:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 07:06:26.557423 14330 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 07:06:26.595767 15012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:26.571388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34632 (local address 127.21.83.2:37843)
0807 07:06:26.585313 (+ 13925us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:26.585324 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:26.586781 (+ 1457us) server_negotiation.cc:407] Connection header received
0807 07:06:26.586892 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:26.586899 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:26.587247 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:26.587441 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:06:26.588781 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:26.590785 (+ 2004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:26.593476 (+ 2691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:26.594532 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:26.594668 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:26.594895 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:26.595228 (+ 333us) server_negotiation.cc:299] Negotiation successful
0807 07:06:26.595418 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13691,"thread_start_us":13951,"threads_started":1}
I20260807 07:06:26.595857 15011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:26.570881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:32999 (local address 127.0.0.1:40014)
0807 07:06:26.582544 (+ 11663us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:26.582631 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0807 07:06:26.582900 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:06:26.587122 (+ 4222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:06:26.587130 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:06:26.587162 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:06:26.587685 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:26.587706 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:26.589733 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:26.589738 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:26.592383 (+ 2645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:26.592400 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:26.595062 (+ 2662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:26.595067 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:26.595252 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:26.595286 (+ 34us) client_negotiation.cc:769] Sending connection context
0807 07:06:26.595366 (+ 80us) client_negotiation.cc:240] Negotiation successful
0807 07:06:26.595488 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11259,"thread_start_us":15268,"threads_started":1}
I20260807 07:06:26.595882 15008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:26.571104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.2:37843 (local address 127.0.0.1:34632)
0807 07:06:26.586453 (+ 15349us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:26.586475 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 07:06:26.586638 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:06:26.587706 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:06:26.587711 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:06:26.587741 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:06:26.588563 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:26.588578 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:26.590946 (+ 2368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:26.590951 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:26.593288 (+ 2337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:26.593301 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:26.594727 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:26.594734 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:26.594938 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:26.594997 (+ 59us) client_negotiation.cc:769] Sending connection context
0807 07:06:26.595169 (+ 172us) client_negotiation.cc:240] Negotiation successful
0807 07:06:26.595383 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15213,"mutex_wait_us":2,"thread_start_us":10833,"threads_started":1}
I20260807 07:06:26.597008 15009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:26.571022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40014 (local address 127.21.83.1:32999)
0807 07:06:26.586105 (+ 15083us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:26.586113 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:26.586148 (+ 35us) server_negotiation.cc:407] Connection header received
0807 07:06:26.586268 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:26.586278 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:26.586576 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:26.586793 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:06:26.587914 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:26.589560 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:26.593127 (+ 3567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:26.594677 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:26.594804 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:26.595057 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:26.595998 (+ 941us) server_negotiation.cc:299] Negotiation successful
0807 07:06:26.596615 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14909,"thread_start_us":14749,"threads_started":1}
I20260807 07:06:26.599983 15013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:26.574137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.3:38789 (local address 127.0.0.1:56624)
0807 07:06:26.591197 (+ 17060us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:26.591224 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0807 07:06:26.591382 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:06:26.592568 (+ 1186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:06:26.592573 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:06:26.592601 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:06:26.593113 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:26.593128 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:26.595039 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:26.595045 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:26.597120 (+ 2075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:26.597136 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:26.599103 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:26.599110 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:26.599287 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:26.599321 (+ 34us) client_negotiation.cc:769] Sending connection context
0807 07:06:26.599392 (+ 71us) client_negotiation.cc:240] Negotiation successful
0807 07:06:26.599549 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16887,"mutex_wait_us":1,"thread_start_us":16519,"threads_started":1}
I20260807 07:06:26.601207 15014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:26.574361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56624 (local address 127.21.83.3:38789)
0807 07:06:26.592077 (+ 17716us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:26.592084 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:26.592113 (+ 29us) server_negotiation.cc:407] Connection header received
0807 07:06:26.592195 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:26.592200 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:26.592396 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:26.592584 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:06:26.593348 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:26.594862 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:26.597378 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:26.598840 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:26.600331 (+ 1491us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:26.600578 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:26.600652 (+ 74us) server_negotiation.cc:299] Negotiation successful
0807 07:06:26.600827 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17543,"thread_start_us":18430,"threads_started":1}
I20260807 07:06:26.600785 14506 tablet_service.cc:1467] Processing CreateTablet for tablet 18b42d8c41994010b07a818215730877 (DEFAULT_TABLE table=test-table [id=940cef4ed51e400abcdea52d1e109960]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:06:26.600744 14698 tablet_service.cc:1467] Processing CreateTablet for tablet 18b42d8c41994010b07a818215730877 (DEFAULT_TABLE table=test-table [id=940cef4ed51e400abcdea52d1e109960]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:06:26.602339 14698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18b42d8c41994010b07a818215730877. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:06:26.602375 14506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18b42d8c41994010b07a818215730877. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:06:26.605444 14890 tablet_service.cc:1467] Processing CreateTablet for tablet 18b42d8c41994010b07a818215730877 (DEFAULT_TABLE table=test-table [id=940cef4ed51e400abcdea52d1e109960]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:06:26.606972 14890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18b42d8c41994010b07a818215730877. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:06:26.614675 15015 tablet_bootstrap.cc:492] T 18b42d8c41994010b07a818215730877 P c135d77f3d784aff8092d220514de9d3: Bootstrap starting.
I20260807 07:06:26.617311 15016 tablet_bootstrap.cc:492] T 18b42d8c41994010b07a818215730877 P 0c4a23cbb00947848c8c0d5ed8b82236: Bootstrap starting.
I20260807 07:06:26.618069 15015 tablet_bootstrap.cc:654] T 18b42d8c41994010b07a818215730877 P c135d77f3d784aff8092d220514de9d3: Neither blocks nor log segments found. Creating new log.
I20260807 07:06:26.619462 15015 log.cc:824] T 18b42d8c41994010b07a818215730877 P c135d77f3d784aff8092d220514de9d3: Log is configured to *not* fsync() on all Append() calls
I20260807 07:06:26.620783 15017 tablet_bootstrap.cc:492] T 18b42d8c41994010b07a818215730877 P 64ece599d1ff40698e76fb7377fe4d9a: Bootstrap starting.
I20260807 07:06:26.621054 15016 tablet_bootstrap.cc:654] T 18b42d8c41994010b07a818215730877 P 0c4a23cbb00947848c8c0d5ed8b82236: Neither blocks nor log segments found. Creating new log.
I20260807 07:06:26.622315 15016 log.cc:824] T 18b42d8c41994010b07a818215730877 P 0c4a23cbb00947848c8c0d5ed8b82236: Log is configured to *not* fsync() on all Append() calls
I20260807 07:06:26.624029 15017 tablet_bootstrap.cc:654] T 18b42d8c41994010b07a818215730877 P 64ece599d1ff40698e76fb7377fe4d9a: Neither blocks nor log segments found. Creating new log.
I20260807 07:06:26.624243 15015 tablet_bootstrap.cc:492] T 18b42d8c41994010b07a818215730877 P c135d77f3d784aff8092d220514de9d3: No bootstrap required, opened a new log
I20260807 07:06:26.624553 15015 ts_tablet_manager.cc:1397] T 18b42d8c41994010b07a818215730877 P c135d77f3d784aff8092d220514de9d3: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260807 07:06:26.625146 15017 log.cc:824] T 18b42d8c41994010b07a818215730877 P 64ece599d1ff40698e76fb7377fe4d9a: Log is configured to *not* fsync() on all Append() calls
I20260807 07:06:26.625331 15016 tablet_bootstrap.cc:492] T 18b42d8c41994010b07a818215730877 P 0c4a23cbb00947848c8c0d5ed8b82236: No bootstrap required, opened a new log
I20260807 07:06:26.625597 15016 ts_tablet_manager.cc:1397] T 18b42d8c41994010b07a818215730877 P 0c4a23cbb00947848c8c0d5ed8b82236: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260807 07:06:26.627481 15017 tablet_bootstrap.cc:492] T 18b42d8c41994010b07a818215730877 P 64ece599d1ff40698e76fb7377fe4d9a: No bootstrap required, opened a new log
I20260807 07:06:26.627735 15017 ts_tablet_manager.cc:1397] T 18b42d8c41994010b07a818215730877 P 64ece599d1ff40698e76fb7377fe4d9a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260807 07:06:26.635177 15017 raft_consensus.cc:348] T 18b42d8c41994010b07a818215730877 P 64ece599d1ff40698e76fb7377fe4d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4a23cbb00947848c8c0d5ed8b82236" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 32999 } } peers { permanent_uuid: "c135d77f3d784aff8092d220514de9d3" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 37843 } } peers { permanent_uuid: "64ece599d1ff40698e76fb7377fe4d9a" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 38789 } }
I20260807 07:06:26.635533 15017 raft_consensus.cc:374] T 18b42d8c41994010b07a818215730877 P 64ece599d1ff40698e76fb7377fe4d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:06:26.635244 15015 raft_consensus.cc:348] T 18b42d8c41994010b07a818215730877 P c135d77f3d784aff8092d220514de9d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4a23cbb00947848c8c0d5ed8b82236" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 32999 } } peers { permanent_uuid: "c135d77f3d784aff8092d220514de9d3" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 37843 } } peers { permanent_uuid: "64ece599d1ff40698e76fb7377fe4d9a" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 38789 } }
I20260807 07:06:26.635639 15017 raft_consensus.cc:729] T 18b42d8c41994010b07a818215730877 P 64ece599d1ff40698e76fb7377fe4d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64ece599d1ff40698e76fb7377fe4d9a, State: Initialized, Role: FOLLOWER
I20260807 07:06:26.635741 15015 raft_consensus.cc:374] T 18b42d8c41994010b07a818215730877 P c135d77f3d784aff8092d220514de9d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:06:26.635851 15015 raft_consensus.cc:729] T 18b42d8c41994010b07a818215730877 P c135d77f3d784aff8092d220514de9d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c135d77f3d784aff8092d220514de9d3, State: Initialized, Role: FOLLOWER
I20260807 07:06:26.635965 15016 raft_consensus.cc:348] T 18b42d8c41994010b07a818215730877 P 0c4a23cbb00947848c8c0d5ed8b82236 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4a23cbb00947848c8c0d5ed8b82236" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 32999 } } peers { permanent_uuid: "c135d77f3d784aff8092d220514de9d3" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 37843 } } peers { permanent_uuid: "64ece599d1ff40698e76fb7377fe4d9a" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 38789 } }
I20260807 07:06:26.636406 15016 raft_consensus.cc:374] T 18b42d8c41994010b07a818215730877 P 0c4a23cbb00947848c8c0d5ed8b82236 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:06:26.636278 15017 consensus_queue.cc:260] T 18b42d8c41994010b07a818215730877 P 64ece599d1ff40698e76fb7377fe4d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4a23cbb00947848c8c0d5ed8b82236" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 32999 } } peers { permanent_uuid: "c135d77f3d784aff8092d220514de9d3" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 37843 } } peers { permanent_uuid: "64ece599d1ff40698e76fb7377fe4d9a" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 38789 } }
I20260807 07:06:26.636510 15016 raft_consensus.cc:729] T 18b42d8c41994010b07a818215730877 P 0c4a23cbb00947848c8c0d5ed8b82236 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c4a23cbb00947848c8c0d5ed8b82236, State: Initialized, Role: FOLLOWER
I20260807 07:06:26.636521 15015 consensus_queue.cc:260] T 18b42d8c41994010b07a818215730877 P c135d77f3d784aff8092d220514de9d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4a23cbb00947848c8c0d5ed8b82236" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 32999 } } peers { permanent_uuid: "c135d77f3d784aff8092d220514de9d3" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 37843 } } peers { permanent_uuid: "64ece599d1ff40698e76fb7377fe4d9a" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 38789 } }
I20260807 07:06:26.637252 15016 consensus_queue.cc:260] T 18b42d8c41994010b07a818215730877 P 0c4a23cbb00947848c8c0d5ed8b82236 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4a23cbb00947848c8c0d5ed8b82236" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 32999 } } peers { permanent_uuid: "c135d77f3d784aff8092d220514de9d3" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 37843 } } peers { permanent_uuid: "64ece599d1ff40698e76fb7377fe4d9a" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 38789 } }
I20260807 07:06:26.637426 14990 heartbeater.cc:503] Master 127.21.83.62:43873 was elected leader, sending a full tablet report...
I20260807 07:06:26.638041 15017 ts_tablet_manager.cc:1428] T 18b42d8c41994010b07a818215730877 P 64ece599d1ff40698e76fb7377fe4d9a: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260807 07:06:26.638919 15016 ts_tablet_manager.cc:1428] T 18b42d8c41994010b07a818215730877 P 0c4a23cbb00947848c8c0d5ed8b82236: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260807 07:06:26.639886 14793 heartbeater.cc:503] Master 127.21.83.62:43873 was elected leader, sending a full tablet report...
I20260807 07:06:26.640589 15015 ts_tablet_manager.cc:1428] T 18b42d8c41994010b07a818215730877 P c135d77f3d784aff8092d220514de9d3: Time spent starting tablet: real 0.016s user 0.013s sys 0.000s
W20260807 07:06:26.688052 14794 tablet.cc:2369] T 18b42d8c41994010b07a818215730877 P c135d77f3d784aff8092d220514de9d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 07:06:26.741562 14991 tablet.cc:2369] T 18b42d8c41994010b07a818215730877 P 64ece599d1ff40698e76fb7377fe4d9a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 07:06:26.770771 15027 raft_consensus.cc:482] T 18b42d8c41994010b07a818215730877 P 0c4a23cbb00947848c8c0d5ed8b82236 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:06:26.771091 15027 raft_consensus.cc:504] T 18b42d8c41994010b07a818215730877 P 0c4a23cbb00947848c8c0d5ed8b82236 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4a23cbb00947848c8c0d5ed8b82236" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 32999 } } peers { permanent_uuid: "c135d77f3d784aff8092d220514de9d3" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 37843 } } peers { permanent_uuid: "64ece599d1ff40698e76fb7377fe4d9a" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 38789 } }
I20260807 07:06:26.772850 15027 leader_election.cc:288] T 18b42d8c41994010b07a818215730877 P 0c4a23cbb00947848c8c0d5ed8b82236 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c135d77f3d784aff8092d220514de9d3 (127.21.83.2:37843), 64ece599d1ff40698e76fb7377fe4d9a (127.21.83.3:38789)
I20260807 07:06:26.781656 15012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:26.773436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:46405 (local address 127.21.83.2:37843)
0807 07:06:26.773660 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:26.773667 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:26.774206 (+ 539us) server_negotiation.cc:407] Connection header received
0807 07:06:26.774354 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:26.774359 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:26.774429 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:26.774524 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:06:26.775284 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:26.776404 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:26.779253 (+ 2849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:26.780510 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:26.780638 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:26.780745 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:26.781239 (+ 494us) server_negotiation.cc:299] Negotiation successful
0807 07:06:26.781379 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 07:06:26.781653 15014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:26.773376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:54743 (local address 127.21.83.3:38789)
0807 07:06:26.773735 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:26.773740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:26.774079 (+ 339us) server_negotiation.cc:407] Connection header received
0807 07:06:26.774335 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:26.774341 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:26.774424 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:26.774532 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:06:26.775317 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:26.776404 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:26.779253 (+ 2849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:26.780509 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:26.780639 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:26.780745 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:26.781239 (+ 494us) server_negotiation.cc:299] Negotiation successful
0807 07:06:26.781379 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194}
I20260807 07:06:26.781869 15029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:26.773269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.2:37843 (local address 127.21.83.1:46405)
0807 07:06:26.774103 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:26.774117 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 07:06:26.774216 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:06:26.774548 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:06:26.774553 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:06:26.774592 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:06:26.775088 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:26.775105 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:26.776550 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:26.776555 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:26.779066 (+ 2511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:26.779083 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:26.780754 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:26.780764 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:26.781025 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:26.781089 (+ 64us) client_negotiation.cc:769] Sending connection context
0807 07:06:26.781179 (+ 90us) client_negotiation.cc:240] Negotiation successful
0807 07:06:26.781366 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"mutex_wait_us":120,"spinlock_wait_cycles":2304,"thread_start_us":557,"threads_started":1}
I20260807 07:06:26.781919 15028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:26.773210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.3:38789 (local address 127.21.83.1:54743)
0807 07:06:26.773963 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:26.773996 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0807 07:06:26.774182 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:06:26.774548 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:06:26.774553 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:06:26.774600 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:06:26.775176 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:26.775187 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:26.776554 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:26.776558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:26.779066 (+ 2508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:26.779083 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:26.780778 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:26.780784 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:26.781031 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:26.781086 (+ 55us) client_negotiation.cc:769] Sending connection context
0807 07:06:26.781191 (+ 105us) client_negotiation.cc:240] Negotiation successful
0807 07:06:26.781366 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"mutex_wait_us":220,"spinlock_wait_cycles":13312,"thread_start_us":381,"threads_started":1}
I20260807 07:06:26.783171 14918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18b42d8c41994010b07a818215730877" candidate_uuid: "0c4a23cbb00947848c8c0d5ed8b82236" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64ece599d1ff40698e76fb7377fe4d9a" is_pre_election: true
I20260807 07:06:26.783250 14725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18b42d8c41994010b07a818215730877" candidate_uuid: "0c4a23cbb00947848c8c0d5ed8b82236" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c135d77f3d784aff8092d220514de9d3" is_pre_election: true
I20260807 07:06:26.783761 14918 raft_consensus.cc:2454] T 18b42d8c41994010b07a818215730877 P 64ece599d1ff40698e76fb7377fe4d9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c4a23cbb00947848c8c0d5ed8b82236 in term 0.
I20260807 07:06:26.783818 14725 raft_consensus.cc:2454] T 18b42d8c41994010b07a818215730877 P c135d77f3d784aff8092d220514de9d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c4a23cbb00947848c8c0d5ed8b82236 in term 0.
I20260807 07:06:26.784466 14446 leader_election.cc:302] T 18b42d8c41994010b07a818215730877 P 0c4a23cbb00947848c8c0d5ed8b82236 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c4a23cbb00947848c8c0d5ed8b82236, c135d77f3d784aff8092d220514de9d3; no voters:
I20260807 07:06:26.784863 15027 raft_consensus.cc:2781] T 18b42d8c41994010b07a818215730877 P 0c4a23cbb00947848c8c0d5ed8b82236 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 07:06:26.785043 15027 raft_consensus.cc:482] T 18b42d8c41994010b07a818215730877 P 0c4a23cbb00947848c8c0d5ed8b82236 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:06:26.785145 15027 raft_consensus.cc:3037] T 18b42d8c41994010b07a818215730877 P 0c4a23cbb00947848c8c0d5ed8b82236 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:06:26.787395 15027 raft_consensus.cc:504] T 18b42d8c41994010b07a818215730877 P 0c4a23cbb00947848c8c0d5ed8b82236 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4a23cbb00947848c8c0d5ed8b82236" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 32999 } } peers { permanent_uuid: "c135d77f3d784aff8092d220514de9d3" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 37843 } } peers { permanent_uuid: "64ece599d1ff40698e76fb7377fe4d9a" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 38789 } }
I20260807 07:06:26.788038 15027 leader_election.cc:288] T 18b42d8c41994010b07a818215730877 P 0c4a23cbb00947848c8c0d5ed8b82236 [CANDIDATE]: Term 1 election: Requested vote from peers c135d77f3d784aff8092d220514de9d3 (127.21.83.2:37843), 64ece599d1ff40698e76fb7377fe4d9a (127.21.83.3:38789)
I20260807 07:06:26.788460 14725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18b42d8c41994010b07a818215730877" candidate_uuid: "0c4a23cbb00947848c8c0d5ed8b82236" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c135d77f3d784aff8092d220514de9d3"
I20260807 07:06:26.788496 14918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18b42d8c41994010b07a818215730877" candidate_uuid: "0c4a23cbb00947848c8c0d5ed8b82236" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64ece599d1ff40698e76fb7377fe4d9a"
I20260807 07:06:26.788661 14725 raft_consensus.cc:3037] T 18b42d8c41994010b07a818215730877 P c135d77f3d784aff8092d220514de9d3 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:06:26.788691 14918 raft_consensus.cc:3037] T 18b42d8c41994010b07a818215730877 P 64ece599d1ff40698e76fb7377fe4d9a [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:06:26.790859 14725 raft_consensus.cc:2454] T 18b42d8c41994010b07a818215730877 P c135d77f3d784aff8092d220514de9d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c4a23cbb00947848c8c0d5ed8b82236 in term 1.
I20260807 07:06:26.790912 14918 raft_consensus.cc:2454] T 18b42d8c41994010b07a818215730877 P 64ece599d1ff40698e76fb7377fe4d9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c4a23cbb00947848c8c0d5ed8b82236 in term 1.
I20260807 07:06:26.791239 14446 leader_election.cc:302] T 18b42d8c41994010b07a818215730877 P 0c4a23cbb00947848c8c0d5ed8b82236 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c4a23cbb00947848c8c0d5ed8b82236, c135d77f3d784aff8092d220514de9d3; no voters:
I20260807 07:06:26.791543 15027 raft_consensus.cc:2781] T 18b42d8c41994010b07a818215730877 P 0c4a23cbb00947848c8c0d5ed8b82236 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:06:26.792023 15027 raft_consensus.cc:686] T 18b42d8c41994010b07a818215730877 P 0c4a23cbb00947848c8c0d5ed8b82236 [term 1 LEADER]: Becoming Leader. State: Replica: 0c4a23cbb00947848c8c0d5ed8b82236, State: Running, Role: LEADER
I20260807 07:06:26.792531 15027 consensus_queue.cc:237] T 18b42d8c41994010b07a818215730877 P 0c4a23cbb00947848c8c0d5ed8b82236 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4a23cbb00947848c8c0d5ed8b82236" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 32999 } } peers { permanent_uuid: "c135d77f3d784aff8092d220514de9d3" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 37843 } } peers { permanent_uuid: "64ece599d1ff40698e76fb7377fe4d9a" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 38789 } }
I20260807 07:06:26.796856 14325 catalog_manager.cc:5495] T 18b42d8c41994010b07a818215730877 P 0c4a23cbb00947848c8c0d5ed8b82236 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c4a23cbb00947848c8c0d5ed8b82236 (127.21.83.1). New cstate: current_term: 1 leader_uuid: "0c4a23cbb00947848c8c0d5ed8b82236" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4a23cbb00947848c8c0d5ed8b82236" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 32999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c135d77f3d784aff8092d220514de9d3" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 37843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64ece599d1ff40698e76fb7377fe4d9a" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 38789 } health_report { overall_health: UNKNOWN } } }
W20260807 07:06:26.840497 14608 tablet.cc:2369] T 18b42d8c41994010b07a818215730877 P 0c4a23cbb00947848c8c0d5ed8b82236: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 07:06:26.841289 15009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:26.835822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40028 (local address 127.21.83.1:32999)
0807 07:06:26.836018 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:26.836022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:26.836042 (+ 20us) server_negotiation.cc:407] Connection header received
0807 07:06:26.836168 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:26.836172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:26.836270 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:26.836394 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:06:26.836705 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:26.837502 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:26.838928 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:26.839340 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:26.839433 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:26.839932 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:06:26.840655 (+ 723us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:06:26.840813 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:26.840926 (+ 113us) server_negotiation.cc:299] Negotiation successful
0807 07:06:26.841021 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 07:06:26.849799 14725 raft_consensus.cc:1261] T 18b42d8c41994010b07a818215730877 P c135d77f3d784aff8092d220514de9d3 [term 1 FOLLOWER]: Refusing update from remote peer 0c4a23cbb00947848c8c0d5ed8b82236: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:06:26.849799 14918 raft_consensus.cc:1261] T 18b42d8c41994010b07a818215730877 P 64ece599d1ff40698e76fb7377fe4d9a [term 1 FOLLOWER]: Refusing update from remote peer 0c4a23cbb00947848c8c0d5ed8b82236: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:06:26.851176 15030 consensus_queue.cc:1035] T 18b42d8c41994010b07a818215730877 P 0c4a23cbb00947848c8c0d5ed8b82236 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64ece599d1ff40698e76fb7377fe4d9a" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 38789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:06:26.851579 15027 consensus_queue.cc:1035] T 18b42d8c41994010b07a818215730877 P 0c4a23cbb00947848c8c0d5ed8b82236 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c135d77f3d784aff8092d220514de9d3" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 37843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:06:26.864871 15037 mvcc.cc:205] Tried to move back new op lower bound from 7315809840525164544 to 7315809840306040832. Current Snapshot: MvccSnapshot[applied={T|T < 7315809840525164544}]
I20260807 07:06:26.875787 14330 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:06:26.876030 14330 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:06:26.879575 14330 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 70c2622b0ea94f86a72dddaee910157c: Sending DeleteTablet for 3 replicas of tablet 18b42d8c41994010b07a818215730877
I20260807 07:06:26.880690 14506 tablet_service.cc:1514] Processing DeleteTablet for tablet 18b42d8c41994010b07a818215730877 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:06:26 UTC) from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:40014
I20260807 07:06:26.880787 14698 tablet_service.cc:1514] Processing DeleteTablet for tablet 18b42d8c41994010b07a818215730877 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:06:26 UTC) from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:34632
I20260807 07:06:26.880985 14890 tablet_service.cc:1514] Processing DeleteTablet for tablet 18b42d8c41994010b07a818215730877 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:06:26 UTC) from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:56624
I20260807 07:06:26.883432 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 14423
I20260807 07:06:26.892411 15052 tablet_replica.cc:330] T 18b42d8c41994010b07a818215730877 P c135d77f3d784aff8092d220514de9d3: stopping tablet replica
I20260807 07:06:26.892745 15054 tablet_replica.cc:330] T 18b42d8c41994010b07a818215730877 P 64ece599d1ff40698e76fb7377fe4d9a: stopping tablet replica
I20260807 07:06:26.892980 15052 raft_consensus.cc:2229] T 18b42d8c41994010b07a818215730877 P c135d77f3d784aff8092d220514de9d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:06:26.893291 15054 raft_consensus.cc:2229] T 18b42d8c41994010b07a818215730877 P 64ece599d1ff40698e76fb7377fe4d9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:06:26.893573 15052 raft_consensus.cc:2258] T 18b42d8c41994010b07a818215730877 P c135d77f3d784aff8092d220514de9d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:06:26.893893 15054 raft_consensus.cc:2258] T 18b42d8c41994010b07a818215730877 P 64ece599d1ff40698e76fb7377fe4d9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:06:26.894734 15051 tablet_replica.cc:330] T 18b42d8c41994010b07a818215730877 P 0c4a23cbb00947848c8c0d5ed8b82236: stopping tablet replica
I20260807 07:06:26.895179 15051 raft_consensus.cc:2229] T 18b42d8c41994010b07a818215730877 P 0c4a23cbb00947848c8c0d5ed8b82236 [term 1 LEADER]: Raft consensus shutting down.
I20260807 07:06:26.895924 15051 raft_consensus.cc:2258] T 18b42d8c41994010b07a818215730877 P 0c4a23cbb00947848c8c0d5ed8b82236 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:06:26.896379 15054 ts_tablet_manager.cc:1905] T 18b42d8c41994010b07a818215730877 P 64ece599d1ff40698e76fb7377fe4d9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:06:26.897631 15052 ts_tablet_manager.cc:1905] T 18b42d8c41994010b07a818215730877 P c135d77f3d784aff8092d220514de9d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:06:26.898376 15051 ts_tablet_manager.cc:1905] T 18b42d8c41994010b07a818215730877 P 0c4a23cbb00947848c8c0d5ed8b82236: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:06:26.903630 15054 ts_tablet_manager.cc:1918] T 18b42d8c41994010b07a818215730877 P 64ece599d1ff40698e76fb7377fe4d9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:06:26.903833 15054 log.cc:1196] T 18b42d8c41994010b07a818215730877 P 64ece599d1ff40698e76fb7377fe4d9a: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-2/wal/wals/18b42d8c41994010b07a818215730877
I20260807 07:06:26.904446 15054 ts_tablet_manager.cc:1939] T 18b42d8c41994010b07a818215730877 P 64ece599d1ff40698e76fb7377fe4d9a: Deleting consensus metadata
I20260807 07:06:26.904618 15052 ts_tablet_manager.cc:1918] T 18b42d8c41994010b07a818215730877 P c135d77f3d784aff8092d220514de9d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:06:26.904819 15052 log.cc:1196] T 18b42d8c41994010b07a818215730877 P c135d77f3d784aff8092d220514de9d3: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-1/wal/wals/18b42d8c41994010b07a818215730877
I20260807 07:06:26.905450 15052 ts_tablet_manager.cc:1939] T 18b42d8c41994010b07a818215730877 P c135d77f3d784aff8092d220514de9d3: Deleting consensus metadata
I20260807 07:06:26.906502 14307 catalog_manager.cc:4841] TS 64ece599d1ff40698e76fb7377fe4d9a (127.21.83.3:38789): tablet 18b42d8c41994010b07a818215730877 (table test-table [id=940cef4ed51e400abcdea52d1e109960]) successfully deleted
I20260807 07:06:26.907286 14308 catalog_manager.cc:4841] TS c135d77f3d784aff8092d220514de9d3 (127.21.83.2:37843): tablet 18b42d8c41994010b07a818215730877 (table test-table [id=940cef4ed51e400abcdea52d1e109960]) successfully deleted
I20260807 07:06:26.908027 15051 ts_tablet_manager.cc:1918] T 18b42d8c41994010b07a818215730877 P 0c4a23cbb00947848c8c0d5ed8b82236: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:06:26.908209 15051 log.cc:1196] T 18b42d8c41994010b07a818215730877 P 0c4a23cbb00947848c8c0d5ed8b82236: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786086078080656-21836-0/minicluster-data/ts-0/wal/wals/18b42d8c41994010b07a818215730877
I20260807 07:06:26.908784 15051 ts_tablet_manager.cc:1939] T 18b42d8c41994010b07a818215730877 P 0c4a23cbb00947848c8c0d5ed8b82236: Deleting consensus metadata
Aug 07 07:06:26 dist-test-slave-ncp9 krb5kdc[14265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.1@KRBTEST.COM
I20260807 07:06:26.910450 14306 catalog_manager.cc:4841] TS 0c4a23cbb00947848c8c0d5ed8b82236 (127.21.83.1:32999): tablet 18b42d8c41994010b07a818215730877 (table test-table [id=940cef4ed51e400abcdea52d1e109960]) successfully deleted
I20260807 07:06:26.916846 15009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:26.885229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40032 (local address 127.21.83.1:32999)
0807 07:06:26.885418 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:26.885423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:26.899726 (+ 14303us) server_negotiation.cc:407] Connection header received
0807 07:06:26.900233 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:26.900239 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:26.900336 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:26.900414 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:26.902275 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:26.903511 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:26.905821 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:26.906458 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:26.906575 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:26.911190 (+ 4615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:26.911201 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:26.911215 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:06:26.911255 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:26.914142 (+ 2887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:26.914825 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:26.914832 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:26.914834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:26.914971 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:26.915328 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:26.915331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:26.915333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:26.915710 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:26.915990 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:26.916293 (+ 303us) server_negotiation.cc:299] Negotiation successful
0807 07:06:26.916536 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 07:06:26.917562 14601 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:06:27.031615 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 14423
I20260807 07:06:27.058868 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 14613
Aug 07 07:06:27 dist-test-slave-ncp9 krb5kdc[14265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.2@KRBTEST.COM
I20260807 07:06:27.072512 15012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:27.059703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34638 (local address 127.21.83.2:37843)
0807 07:06:27.059863 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:27.059867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:27.059885 (+ 18us) server_negotiation.cc:407] Connection header received
0807 07:06:27.059955 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:27.059959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:27.060039 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:27.060115 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:27.061375 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:27.062103 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:27.063272 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:27.063644 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:27.063734 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:27.067569 (+ 3835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:27.067583 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:27.067597 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:06:27.067636 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:27.070282 (+ 2646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:27.070904 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:27.070910 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:27.070912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:27.071039 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:27.071339 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:27.071341 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:27.071343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:27.071702 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:27.071913 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:27.072092 (+ 179us) server_negotiation.cc:299] Negotiation successful
0807 07:06:27.072264 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260807 07:06:27.073170 14786 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:06:27.164331 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 14613
I20260807 07:06:27.192653 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 14800
Aug 07 07:06:27 dist-test-slave-ncp9 krb5kdc[14265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.3@KRBTEST.COM
I20260807 07:06:27.207015 15014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:27.193441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56632 (local address 127.21.83.3:38789)
0807 07:06:27.193633 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:27.193637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:27.193655 (+ 18us) server_negotiation.cc:407] Connection header received
0807 07:06:27.193767 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:27.193771 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:27.193846 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:27.193925 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:27.195202 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:27.195910 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:27.197073 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:27.197484 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:27.197575 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:27.201845 (+ 4270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:27.201859 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:27.201875 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:06:27.201908 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:27.204558 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:27.205252 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:27.205266 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:27.205268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:27.205425 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:27.205744 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:27.205748 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:27.205749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:27.206086 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:27.206303 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:27.206579 (+ 276us) server_negotiation.cc:299] Negotiation successful
0807 07:06:27.206755 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260807 07:06:27.207626 14984 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:06:27.297981 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 14800
I20260807 07:06:27.322057 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 14283
I20260807 07:06:27.323239 14379 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:06:27.435734 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 14283
2026-08-07T07:06:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4847 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-07T07:06:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:06:27Z Disabled control of system clock
I20260807 07:06:27.550515 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:39941
--webserver_interface=127.21.83.62
--webserver_port=0
--builtin_ntp_servers=127.21.83.20:37571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:39941
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 07:06:27.903589 15066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:06:27.903915 15066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:06:27.904016 15066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:06:27.911188 15066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:06:27.911294 15066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:06:27.911339 15066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:06:27.911396 15066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:06:27.911438 15066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:06:27.920509 15066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:37571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:39941
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:39941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:06:27.921766 15066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:06:27.923485 15066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:06:27.934589 15075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:27.934676 15079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:27.934908 15076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:06:27.936028 15066 server_base.cc:1034] running on GCE node
I20260807 07:06:27.936741 15066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:06:27.938079 15066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:06:27.939282 15066 hybrid_clock.cc:648] HybridClock initialized: now 1786086387939240 us; error 30 us; skew 500 ppm
I20260807 07:06:27.939607 15066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 07:06:27.947031 15066 webserver.cc:460] Webserver started at http://127.21.83.62:35365/ using document root <none> and password file <none>
I20260807 07:06:27.947849 15066 fs_manager.cc:359] Metadata directory not provided
I20260807 07:06:27.947978 15066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:06:27.948336 15066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:06:27.951625 15066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "b57daf68acbc449b916ec0d167b3c6a7"
format_stamp: "Formatted at 2026-08-07 07:06:27 on dist-test-slave-ncp9"
server_key: "7d3de938f1156cc54a76cc9869a4668a"
server_key_iv: "c1560e8eb177fe743857c8a3512fff8a"
server_key_version: "encryptionkey@0"
I20260807 07:06:27.952515 15066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "b57daf68acbc449b916ec0d167b3c6a7"
format_stamp: "Formatted at 2026-08-07 07:06:27 on dist-test-slave-ncp9"
server_key: "7d3de938f1156cc54a76cc9869a4668a"
server_key_iv: "c1560e8eb177fe743857c8a3512fff8a"
server_key_version: "encryptionkey@0"
I20260807 07:06:27.958473 15066 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20260807 07:06:27.962100 15092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:27.963408 15066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260807 07:06:27.963587 15066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "b57daf68acbc449b916ec0d167b3c6a7"
format_stamp: "Formatted at 2026-08-07 07:06:27 on dist-test-slave-ncp9"
server_key: "7d3de938f1156cc54a76cc9869a4668a"
server_key_iv: "c1560e8eb177fe743857c8a3512fff8a"
server_key_version: "encryptionkey@0"
I20260807 07:06:27.963814 15066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:06:27.975883 15066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:06:27.976627 15066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:06:27.976886 15066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:06:27.994573 15066 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.62:39941
I20260807 07:06:27.994624 15164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.62:39941 every 8 connection(s)
I20260807 07:06:27.996073 15066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:06:28.000185 15165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:06:28.005954 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 15066
I20260807 07:06:28.006135 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:06:28.006594 21836 external_mini_cluster.cc:1428] Setting key 5717c312db3f46ef605ce6b2438e4ca0
I20260807 07:06:28.012600 15165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b57daf68acbc449b916ec0d167b3c6a7: Bootstrap starting.
I20260807 07:06:28.016510 15165 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b57daf68acbc449b916ec0d167b3c6a7: Neither blocks nor log segments found. Creating new log.
I20260807 07:06:28.017669 15165 log.cc:824] T 00000000000000000000000000000000 P b57daf68acbc449b916ec0d167b3c6a7: Log is configured to *not* fsync() on all Append() calls
I20260807 07:06:28.021215 15165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b57daf68acbc449b916ec0d167b3c6a7: No bootstrap required, opened a new log
I20260807 07:06:28.022341 15169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:28.008652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37106 (local address 127.21.83.62:39941)
0807 07:06:28.009824 (+ 1172us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:28.009842 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:28.009900 (+ 58us) server_negotiation.cc:407] Connection header received
0807 07:06:28.012230 (+ 2330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:28.012260 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:28.012874 (+ 614us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:28.013334 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:28.013870 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:28.015828 (+ 1958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:28.017254 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:28.018293 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:28.018467 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:28.019668 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:28.019680 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:28.019697 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:06:28.020367 (+ 670us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:28.020464 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:06:28.020491 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:28.020630 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:28.020762 (+ 132us) server_negotiation.cc:299] Negotiation successful
0807 07:06:28.021106 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":387,"threads_started":1}
I20260807 07:06:28.029259 15165 raft_consensus.cc:348] T 00000000000000000000000000000000 P b57daf68acbc449b916ec0d167b3c6a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57daf68acbc449b916ec0d167b3c6a7" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39941 } }
I20260807 07:06:28.029675 15165 raft_consensus.cc:374] T 00000000000000000000000000000000 P b57daf68acbc449b916ec0d167b3c6a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:06:28.029763 15165 raft_consensus.cc:729] T 00000000000000000000000000000000 P b57daf68acbc449b916ec0d167b3c6a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b57daf68acbc449b916ec0d167b3c6a7, State: Initialized, Role: FOLLOWER
I20260807 07:06:28.030246 15165 consensus_queue.cc:260] T 00000000000000000000000000000000 P b57daf68acbc449b916ec0d167b3c6a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57daf68acbc449b916ec0d167b3c6a7" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39941 } }
I20260807 07:06:28.030454 15165 raft_consensus.cc:388] T 00000000000000000000000000000000 P b57daf68acbc449b916ec0d167b3c6a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:06:28.030550 15165 raft_consensus.cc:482] T 00000000000000000000000000000000 P b57daf68acbc449b916ec0d167b3c6a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:06:28.030676 15165 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b57daf68acbc449b916ec0d167b3c6a7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:06:28.032795 15165 raft_consensus.cc:504] T 00000000000000000000000000000000 P b57daf68acbc449b916ec0d167b3c6a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57daf68acbc449b916ec0d167b3c6a7" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39941 } }
I20260807 07:06:28.033260 15165 leader_election.cc:302] T 00000000000000000000000000000000 P b57daf68acbc449b916ec0d167b3c6a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b57daf68acbc449b916ec0d167b3c6a7; no voters:
I20260807 07:06:28.033793 15165 leader_election.cc:288] T 00000000000000000000000000000000 P b57daf68acbc449b916ec0d167b3c6a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 07:06:28.033887 15171 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b57daf68acbc449b916ec0d167b3c6a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:06:28.035055 15171 raft_consensus.cc:686] T 00000000000000000000000000000000 P b57daf68acbc449b916ec0d167b3c6a7 [term 1 LEADER]: Becoming Leader. State: Replica: b57daf68acbc449b916ec0d167b3c6a7, State: Running, Role: LEADER
I20260807 07:06:28.035518 15171 consensus_queue.cc:237] T 00000000000000000000000000000000 P b57daf68acbc449b916ec0d167b3c6a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57daf68acbc449b916ec0d167b3c6a7" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39941 } }
I20260807 07:06:28.035878 15165 sys_catalog.cc:564] T 00000000000000000000000000000000 P b57daf68acbc449b916ec0d167b3c6a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:06:28.038172 15175 sys_catalog.cc:455] T 00000000000000000000000000000000 P b57daf68acbc449b916ec0d167b3c6a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b57daf68acbc449b916ec0d167b3c6a7. Latest consensus state: current_term: 1 leader_uuid: "b57daf68acbc449b916ec0d167b3c6a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57daf68acbc449b916ec0d167b3c6a7" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39941 } } }
I20260807 07:06:28.038586 15175 sys_catalog.cc:458] T 00000000000000000000000000000000 P b57daf68acbc449b916ec0d167b3c6a7 [sys.catalog]: This master's current role is: LEADER
I20260807 07:06:28.039093 15174 sys_catalog.cc:455] T 00000000000000000000000000000000 P b57daf68acbc449b916ec0d167b3c6a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b57daf68acbc449b916ec0d167b3c6a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57daf68acbc449b916ec0d167b3c6a7" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39941 } } }
I20260807 07:06:28.039369 15174 sys_catalog.cc:458] T 00000000000000000000000000000000 P b57daf68acbc449b916ec0d167b3c6a7 [sys.catalog]: This master's current role is: LEADER
I20260807 07:06:28.040373 15181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:06:28.045416 15181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:06:28.054802 15181 catalog_manager.cc:1347] Generated new cluster ID: 16d13eb1763540c08886b1899a70956b
I20260807 07:06:28.054934 15181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:06:28.083036 15181 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:06:28.084954 15181 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:06:28.094520 15181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b57daf68acbc449b916ec0d167b3c6a7: Generated new TSK 0
I20260807 07:06:28.095352 15181 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 07:06:28.112422 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.1:0
--local_ip_for_outbound_sockets=127.21.83.1
--webserver_interface=127.21.83.1
--webserver_port=0
--tserver_master_addrs=127.21.83.62:39941
--builtin_ntp_servers=127.21.83.20:37571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 07:06:28.477058 15198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:06:28.477312 15198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:06:28.477406 15198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:06:28.484581 15198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:06:28.484664 15198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:06:28.484763 15198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.1
I20260807 07:06:28.493381 15198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:37571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.83.1
--webserver_port=0
--tserver_master_addrs=127.21.83.62:39941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.1
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:06:28.494469 15198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:06:28.496136 15198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:06:28.508002 15206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:28.508059 15203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:28.508097 15204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:06:28.508915 15198 server_base.cc:1034] running on GCE node
I20260807 07:06:28.509552 15198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:06:28.510562 15198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:06:28.511828 15198 hybrid_clock.cc:648] HybridClock initialized: now 1786086388511732 us; error 84 us; skew 500 ppm
I20260807 07:06:28.512188 15198 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 07:06:28.514686 15198 webserver.cc:460] Webserver started at http://127.21.83.1:42363/ using document root <none> and password file <none>
I20260807 07:06:28.515280 15198 fs_manager.cc:359] Metadata directory not provided
I20260807 07:06:28.515363 15198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:06:28.515625 15198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:06:28.518394 15198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-0/data/instance:
uuid: "d4e4ede98de94f9d893305084c3926dc"
format_stamp: "Formatted at 2026-08-07 07:06:28 on dist-test-slave-ncp9"
server_key: "ad45cf622c64daa2ccaef74d8b65327a"
server_key_iv: "1c0ea2acfd61cde8ce851a8c21a8c62c"
server_key_version: "encryptionkey@0"
I20260807 07:06:28.519150 15198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-0/wal/instance:
uuid: "d4e4ede98de94f9d893305084c3926dc"
format_stamp: "Formatted at 2026-08-07 07:06:28 on dist-test-slave-ncp9"
server_key: "ad45cf622c64daa2ccaef74d8b65327a"
server_key_iv: "1c0ea2acfd61cde8ce851a8c21a8c62c"
server_key_version: "encryptionkey@0"
I20260807 07:06:28.524471 15198 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 07:06:28.527781 15219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:28.529021 15198 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 07:06:28.529172 15198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-0/wal
uuid: "d4e4ede98de94f9d893305084c3926dc"
format_stamp: "Formatted at 2026-08-07 07:06:28 on dist-test-slave-ncp9"
server_key: "ad45cf622c64daa2ccaef74d8b65327a"
server_key_iv: "1c0ea2acfd61cde8ce851a8c21a8c62c"
server_key_version: "encryptionkey@0"
I20260807 07:06:28.529366 15198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:06:28.550354 15198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:06:28.551318 15198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:06:28.551609 15198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:06:28.552716 15198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:06:28.554355 15198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:06:28.554463 15198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:28.554574 15198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:06:28.554647 15198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:28.585508 15198 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.1:34393
I20260807 07:06:28.585608 15389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.1:34393 every 8 connection(s)
I20260807 07:06:28.586791 15198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
I20260807 07:06:28.592015 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 15198
I20260807 07:06:28.592178 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-0/wal/instance
I20260807 07:06:28.592608 21836 external_mini_cluster.cc:1428] Setting key 876fe548064ef088e684dd67a14f1850
I20260807 07:06:28.595327 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.2:0
--local_ip_for_outbound_sockets=127.21.83.2
--webserver_interface=127.21.83.2
--webserver_port=0
--tserver_master_addrs=127.21.83.62:39941
--builtin_ntp_servers=127.21.83.20:37571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 07:06:28.599351 15394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:28.589266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:56435 (local address 127.21.83.62:39941)
0807 07:06:28.589703 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:28.589707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:28.590822 (+ 1115us) server_negotiation.cc:407] Connection header received
0807 07:06:28.592627 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:28.592632 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:28.592758 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:28.592856 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:28.594020 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:28.594935 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:28.596531 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:28.597177 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:28.597306 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:28.598547 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:28.598553 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:28.598557 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:06:28.598577 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:28.598623 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:06:28.598638 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:28.598723 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:28.598921 (+ 198us) server_negotiation.cc:299] Negotiation successful
0807 07:06:28.599016 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":217,"threads_started":1}
I20260807 07:06:28.600204 15395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:28.589828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39941 (local address 127.21.83.1:56435)
0807 07:06:28.590673 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:28.590718 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 07:06:28.592366 (+ 1648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:06:28.593119 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:06:28.593127 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:06:28.593208 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:06:28.593847 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:28.593866 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:28.595087 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:28.595091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:28.596368 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:28.596381 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:28.597375 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:28.597381 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:28.597595 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:28.598244 (+ 649us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 07:06:28.598246 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:06:28.598283 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 07:06:28.598362 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:06:28.598737 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:06:28.598747 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:06:28.598753 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 07:06:28.598892 (+ 139us) client_negotiation.cc:240] Negotiation successful
0807 07:06:28.599184 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":266,"threads_started":1}
I20260807 07:06:28.601713 15391 heartbeater.cc:347] Connected to a master server at 127.21.83.62:39941
I20260807 07:06:28.602037 15391 heartbeater.cc:464] Registering TS with master...
I20260807 07:06:28.602948 15391 heartbeater.cc:511] Master 127.21.83.62:39941 requested a full tablet report, sending...
I20260807 07:06:28.604995 15116 ts_manager.cc:194] Registered new tserver with Master: d4e4ede98de94f9d893305084c3926dc (127.21.83.1:34393)
I20260807 07:06:28.606381 15116 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.83.1:56435
W20260807 07:06:28.937048 15396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:06:28.937384 15396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:06:28.937531 15396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:06:28.946434 15396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:06:28.946605 15396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:06:28.946763 15396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.2
I20260807 07:06:28.956421 15396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:37571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.83.2
--webserver_port=0
--tserver_master_addrs=127.21.83.62:39941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.2
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:06:28.957851 15396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:06:28.959576 15396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:06:28.972083 15402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:28.972039 15403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:28.971994 15406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:06:28.972846 15396 server_base.cc:1034] running on GCE node
I20260807 07:06:28.973459 15396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:06:28.974385 15396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:06:28.975582 15396 hybrid_clock.cc:648] HybridClock initialized: now 1786086388975534 us; error 35 us; skew 500 ppm
I20260807 07:06:28.975888 15396 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 07:06:28.978193 15396 webserver.cc:460] Webserver started at http://127.21.83.2:33415/ using document root <none> and password file <none>
I20260807 07:06:28.978889 15396 fs_manager.cc:359] Metadata directory not provided
I20260807 07:06:28.978986 15396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:06:28.979331 15396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:06:28.982445 15396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-1/data/instance:
uuid: "48e2225ba70a41389cd3b752f839fb62"
format_stamp: "Formatted at 2026-08-07 07:06:28 on dist-test-slave-ncp9"
server_key: "b0bdc388c9d6ddf8fda6166507c13b29"
server_key_iv: "7207f79c28ed9d8003977c025ec316b6"
server_key_version: "encryptionkey@0"
I20260807 07:06:28.983260 15396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-1/wal/instance:
uuid: "48e2225ba70a41389cd3b752f839fb62"
format_stamp: "Formatted at 2026-08-07 07:06:28 on dist-test-slave-ncp9"
server_key: "b0bdc388c9d6ddf8fda6166507c13b29"
server_key_iv: "7207f79c28ed9d8003977c025ec316b6"
server_key_version: "encryptionkey@0"
I20260807 07:06:28.988431 15396 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 07:06:28.992002 15416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:28.993384 15396 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 07:06:28.993563 15396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-1/wal
uuid: "48e2225ba70a41389cd3b752f839fb62"
format_stamp: "Formatted at 2026-08-07 07:06:28 on dist-test-slave-ncp9"
server_key: "b0bdc388c9d6ddf8fda6166507c13b29"
server_key_iv: "7207f79c28ed9d8003977c025ec316b6"
server_key_version: "encryptionkey@0"
I20260807 07:06:28.993777 15396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:06:29.018929 15396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:06:29.019809 15396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:06:29.020073 15396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:06:29.021242 15396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:06:29.022992 15396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:06:29.023088 15396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:29.023190 15396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:06:29.023249 15396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:29.060036 15396 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.2:33293
I20260807 07:06:29.060138 15584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.2:33293 every 8 connection(s)
I20260807 07:06:29.061367 15396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
I20260807 07:06:29.069674 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 15396
I20260807 07:06:29.069865 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-1/wal/instance
I20260807 07:06:29.070494 21836 external_mini_cluster.cc:1428] Setting key 9a97e9a2e3fcf7d2d78c3c4f2deb1103
I20260807 07:06:29.073652 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.3:0
--local_ip_for_outbound_sockets=127.21.83.3
--webserver_interface=127.21.83.3
--webserver_port=0
--tserver_master_addrs=127.21.83.62:39941
--builtin_ntp_servers=127.21.83.20:37571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 07:06:29.074108 15394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:29.063951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:35189 (local address 127.21.83.62:39941)
0807 07:06:29.064127 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:29.064132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:29.065387 (+ 1255us) server_negotiation.cc:407] Connection header received
0807 07:06:29.067160 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:29.067165 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:29.067245 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:29.067331 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:29.068336 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:29.069140 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:29.071332 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:29.071855 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:29.071982 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:29.073374 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:29.073379 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:29.073383 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:06:29.073401 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:29.073434 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:06:29.073447 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:29.073516 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:29.073764 (+ 248us) server_negotiation.cc:299] Negotiation successful
0807 07:06:29.073874 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 07:06:29.075408 15590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:29.064313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39941 (local address 127.21.83.2:35189)
0807 07:06:29.065236 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:29.065277 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 07:06:29.066907 (+ 1630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:06:29.067507 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:06:29.067531 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:06:29.067620 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:06:29.068183 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:29.068201 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:29.069279 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:29.069284 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:29.071187 (+ 1903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:29.071200 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:29.072059 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:29.072064 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:29.072264 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:29.073026 (+ 762us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 07:06:29.073030 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:06:29.073082 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 07:06:29.073187 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:06:29.073549 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:06:29.073560 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:06:29.073569 (+ 9us) client_negotiation.cc:769] Sending connection context
0807 07:06:29.073759 (+ 190us) client_negotiation.cc:240] Negotiation successful
0807 07:06:29.074143 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":268,"threads_started":1}
I20260807 07:06:29.076800 15585 heartbeater.cc:347] Connected to a master server at 127.21.83.62:39941
I20260807 07:06:29.077136 15585 heartbeater.cc:464] Registering TS with master...
I20260807 07:06:29.077986 15585 heartbeater.cc:511] Master 127.21.83.62:39941 requested a full tablet report, sending...
I20260807 07:06:29.079488 15116 ts_manager.cc:194] Registered new tserver with Master: 48e2225ba70a41389cd3b752f839fb62 (127.21.83.2:33293)
I20260807 07:06:29.080272 15116 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.83.2:35189
W20260807 07:06:29.419447 15591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:06:29.419737 15591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:06:29.419826 15591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:06:29.427080 15591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:06:29.427204 15591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:06:29.427330 15591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.3
I20260807 07:06:29.436697 15591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:37571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.83.3
--webserver_port=0
--tserver_master_addrs=127.21.83.62:39941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.3
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:06:29.438135 15591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:06:29.439986 15591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:06:29.460251 15596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:29.460218 15597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:29.460559 15599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:06:29.461036 15591 server_base.cc:1034] running on GCE node
I20260807 07:06:29.461721 15591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:06:29.462674 15591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:06:29.463871 15591 hybrid_clock.cc:648] HybridClock initialized: now 1786086389463820 us; error 44 us; skew 500 ppm
I20260807 07:06:29.464190 15591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 07:06:29.466424 15591 webserver.cc:460] Webserver started at http://127.21.83.3:36033/ using document root <none> and password file <none>
I20260807 07:06:29.467088 15591 fs_manager.cc:359] Metadata directory not provided
I20260807 07:06:29.467200 15591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:06:29.467474 15591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:06:29.470343 15591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-2/data/instance:
uuid: "72406d752c154c49a65f7f1ccdb95e57"
format_stamp: "Formatted at 2026-08-07 07:06:29 on dist-test-slave-ncp9"
server_key: "a759e390cdbb8858032974e49ef618b9"
server_key_iv: "c3aa8ed5dbfb349075e2fe3752f78c2c"
server_key_version: "encryptionkey@0"
I20260807 07:06:29.471077 15591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-2/wal/instance:
uuid: "72406d752c154c49a65f7f1ccdb95e57"
format_stamp: "Formatted at 2026-08-07 07:06:29 on dist-test-slave-ncp9"
server_key: "a759e390cdbb8858032974e49ef618b9"
server_key_iv: "c3aa8ed5dbfb349075e2fe3752f78c2c"
server_key_version: "encryptionkey@0"
I20260807 07:06:29.475996 15591 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 07:06:29.479215 15614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:29.480377 15591 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 07:06:29.480530 15591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-2/wal
uuid: "72406d752c154c49a65f7f1ccdb95e57"
format_stamp: "Formatted at 2026-08-07 07:06:29 on dist-test-slave-ncp9"
server_key: "a759e390cdbb8858032974e49ef618b9"
server_key_iv: "c3aa8ed5dbfb349075e2fe3752f78c2c"
server_key_version: "encryptionkey@0"
I20260807 07:06:29.480721 15591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:06:29.511368 15591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:06:29.512243 15591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:06:29.512502 15591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:06:29.513628 15591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:06:29.515592 15591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:06:29.515704 15591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:29.515812 15591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:06:29.515878 15591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:29.548157 15591 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.3:33087
I20260807 07:06:29.548221 15763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.3:33087 every 8 connection(s)
I20260807 07:06:29.549618 15591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
I20260807 07:06:29.552949 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 15591
I20260807 07:06:29.553138 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-2/wal/instance
I20260807 07:06:29.553740 21836 external_mini_cluster.cc:1428] Setting key 8d73c9bae791a27229035eceb4dc3293
I20260807 07:06:29.563164 15394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:29.552347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:55035 (local address 127.21.83.62:39941)
0807 07:06:29.552564 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:29.552570 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:29.554266 (+ 1696us) server_negotiation.cc:407] Connection header received
0807 07:06:29.556433 (+ 2167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:29.556438 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:29.556531 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:29.556643 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:29.557858 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:29.558708 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:29.560326 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:29.560735 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:29.560828 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:29.562343 (+ 1515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:29.562349 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:29.562352 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:06:29.562375 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:29.562422 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:06:29.562436 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:29.562522 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:29.562727 (+ 205us) server_negotiation.cc:299] Negotiation successful
0807 07:06:29.562826 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 07:06:29.564150 15767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:29.552822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:39941 (local address 127.21.83.3:55035)
0807 07:06:29.554046 (+ 1224us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:29.554105 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0807 07:06:29.556128 (+ 2023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:06:29.556936 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:06:29.556950 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:06:29.557067 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:06:29.557693 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:29.557711 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:29.558899 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:29.558903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:29.560203 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:29.560217 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:29.561003 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:29.561010 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:29.561234 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:29.561964 (+ 730us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 07:06:29.561966 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:06:29.562012 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 07:06:29.562103 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:06:29.562532 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:06:29.562539 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:06:29.562545 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 07:06:29.562689 (+ 144us) client_negotiation.cc:240] Negotiation successful
0807 07:06:29.563026 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":359,"threads_started":1}
I20260807 07:06:29.565632 15765 heartbeater.cc:347] Connected to a master server at 127.21.83.62:39941
I20260807 07:06:29.565929 15765 heartbeater.cc:464] Registering TS with master...
I20260807 07:06:29.566671 15765 heartbeater.cc:511] Master 127.21.83.62:39941 requested a full tablet report, sending...
I20260807 07:06:29.568348 15115 ts_manager.cc:194] Registered new tserver with Master: 72406d752c154c49a65f7f1ccdb95e57 (127.21.83.3:33087)
I20260807 07:06:29.569315 21836 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 07:06:29.569484 15115 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.83.3:55035
I20260807 07:06:29.580024 15394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:29.573726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58382 (local address 127.21.83.62:39941)
0807 07:06:29.573946 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:29.573952 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:29.574306 (+ 354us) server_negotiation.cc:407] Connection header received
0807 07:06:29.574453 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:29.574457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:29.574540 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:29.574632 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:29.575239 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:29.575962 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:29.577481 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:29.577905 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:29.578018 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:29.579304 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:29.579309 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:29.579312 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:06:29.579334 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:29.579370 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:06:29.579383 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:29.579461 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:29.579601 (+ 140us) server_negotiation.cc:299] Negotiation successful
0807 07:06:29.579717 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260807 07:06:29.584239 15115 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58382:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 07:06:29.591300 15115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 07:06:29.609637 15391 heartbeater.cc:503] Master 127.21.83.62:39941 was elected leader, sending a full tablet report...
I20260807 07:06:29.615154 15786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:29.604089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:34393 (local address 127.0.0.1:58812)
0807 07:06:29.606177 (+ 2088us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:29.606195 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 07:06:29.606323 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:06:29.607345 (+ 1022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:06:29.607353 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:06:29.607386 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:06:29.607783 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:29.607796 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:29.609542 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:29.609546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:29.612131 (+ 2585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:29.612150 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:29.614071 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:29.614077 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:29.614349 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:29.614403 (+ 54us) client_negotiation.cc:769] Sending connection context
0807 07:06:29.614537 (+ 134us) client_negotiation.cc:240] Negotiation successful
0807 07:06:29.614752 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1949,"mutex_wait_us":21,"thread_start_us":1798,"threads_started":1}
I20260807 07:06:29.615145 15789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:29.604622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58812 (local address 127.21.83.1:34393)
0807 07:06:29.605739 (+ 1117us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:29.605748 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:29.606454 (+ 706us) server_negotiation.cc:407] Connection header received
0807 07:06:29.606598 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:29.606607 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:29.606866 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:29.607055 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:06:29.608029 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:29.609363 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:29.612340 (+ 2977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:29.613871 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:29.614016 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:29.614319 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:29.614596 (+ 277us) server_negotiation.cc:299] Negotiation successful
0807 07:06:29.614785 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":920,"thread_start_us":785,"threads_started":1}
I20260807 07:06:29.616281 15782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:29.603737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.2:33293 (local address 127.0.0.1:33114)
0807 07:06:29.604800 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:29.604845 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 07:06:29.605089 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:06:29.608427 (+ 3338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:06:29.608432 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:06:29.608458 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:06:29.608840 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:29.608851 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:29.611372 (+ 2521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:29.611378 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:29.613227 (+ 1849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:29.613244 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:29.615083 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:29.615091 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:29.615731 (+ 640us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:29.615768 (+ 37us) client_negotiation.cc:769] Sending connection context
0807 07:06:29.615832 (+ 64us) client_negotiation.cc:240] Negotiation successful
0807 07:06:29.615967 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"thread_start_us":610,"threads_started":1}
I20260807 07:06:29.617435 15791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:29.605828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37036 (local address 127.21.83.3:33087)
0807 07:06:29.607436 (+ 1608us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:29.607444 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:29.608311 (+ 867us) server_negotiation.cc:407] Connection header received
0807 07:06:29.609073 (+ 762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:29.609078 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:29.609255 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:29.609443 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:06:29.610960 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:29.612429 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:29.614910 (+ 2481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:29.616094 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:29.616216 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:29.616506 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:29.616858 (+ 352us) server_negotiation.cc:299] Negotiation successful
0807 07:06:29.617075 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1429,"thread_start_us":1269,"threads_started":1}
I20260807 07:06:29.617494 15784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:29.604004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33114 (local address 127.21.83.2:33293)
0807 07:06:29.607903 (+ 3899us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:29.607912 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:29.607943 (+ 31us) server_negotiation.cc:407] Connection header received
0807 07:06:29.608037 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:29.608044 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:29.608243 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:29.608437 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:06:29.609738 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:29.611211 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:29.613477 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:29.614857 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:29.616616 (+ 1759us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:29.616850 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:29.616962 (+ 112us) server_negotiation.cc:299] Negotiation successful
0807 07:06:29.617140 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3722,"thread_start_us":3598,"threads_started":1}
I20260807 07:06:29.618088 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:29.605198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.3:33087 (local address 127.0.0.1:37036)
0807 07:06:29.608185 (+ 2987us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:29.608201 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 07:06:29.608321 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:06:29.610268 (+ 1947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:06:29.610272 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:06:29.610302 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:06:29.610723 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:29.610735 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:29.612558 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:29.612563 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:29.614691 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:29.614704 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:29.616298 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:29.616305 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:29.616472 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:29.616504 (+ 32us) client_negotiation.cc:769] Sending connection context
0807 07:06:29.616586 (+ 82us) client_negotiation.cc:240] Negotiation successful
0807 07:06:29.616714 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2875,"mutex_wait_us":26,"thread_start_us":2735,"threads_started":1}
I20260807 07:06:29.619622 15292 tablet_service.cc:1467] Processing CreateTablet for tablet 0b13db9070bd4d548ddd0fb8543bd421 (DEFAULT_TABLE table=test-table [id=1c108a2163e44224a3e0ade2949f8b7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:06:29.621238 15292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b13db9070bd4d548ddd0fb8543bd421. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:06:29.621045 15677 tablet_service.cc:1467] Processing CreateTablet for tablet 0b13db9070bd4d548ddd0fb8543bd421 (DEFAULT_TABLE table=test-table [id=1c108a2163e44224a3e0ade2949f8b7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:06:29.621057 15489 tablet_service.cc:1467] Processing CreateTablet for tablet 0b13db9070bd4d548ddd0fb8543bd421 (DEFAULT_TABLE table=test-table [id=1c108a2163e44224a3e0ade2949f8b7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:06:29.622279 15677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b13db9070bd4d548ddd0fb8543bd421. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:06:29.622383 15489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b13db9070bd4d548ddd0fb8543bd421. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:06:29.634298 15793 tablet_bootstrap.cc:492] T 0b13db9070bd4d548ddd0fb8543bd421 P 72406d752c154c49a65f7f1ccdb95e57: Bootstrap starting.
I20260807 07:06:29.638247 15793 tablet_bootstrap.cc:654] T 0b13db9070bd4d548ddd0fb8543bd421 P 72406d752c154c49a65f7f1ccdb95e57: Neither blocks nor log segments found. Creating new log.
I20260807 07:06:29.639361 15793 log.cc:824] T 0b13db9070bd4d548ddd0fb8543bd421 P 72406d752c154c49a65f7f1ccdb95e57: Log is configured to *not* fsync() on all Append() calls
I20260807 07:06:29.640394 15796 tablet_bootstrap.cc:492] T 0b13db9070bd4d548ddd0fb8543bd421 P 48e2225ba70a41389cd3b752f839fb62: Bootstrap starting.
I20260807 07:06:29.641444 15797 tablet_bootstrap.cc:492] T 0b13db9070bd4d548ddd0fb8543bd421 P d4e4ede98de94f9d893305084c3926dc: Bootstrap starting.
I20260807 07:06:29.644198 15793 tablet_bootstrap.cc:492] T 0b13db9070bd4d548ddd0fb8543bd421 P 72406d752c154c49a65f7f1ccdb95e57: No bootstrap required, opened a new log
I20260807 07:06:29.644490 15793 ts_tablet_manager.cc:1397] T 0b13db9070bd4d548ddd0fb8543bd421 P 72406d752c154c49a65f7f1ccdb95e57: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260807 07:06:29.645645 15797 tablet_bootstrap.cc:654] T 0b13db9070bd4d548ddd0fb8543bd421 P d4e4ede98de94f9d893305084c3926dc: Neither blocks nor log segments found. Creating new log.
I20260807 07:06:29.645645 15796 tablet_bootstrap.cc:654] T 0b13db9070bd4d548ddd0fb8543bd421 P 48e2225ba70a41389cd3b752f839fb62: Neither blocks nor log segments found. Creating new log.
I20260807 07:06:29.647208 15796 log.cc:824] T 0b13db9070bd4d548ddd0fb8543bd421 P 48e2225ba70a41389cd3b752f839fb62: Log is configured to *not* fsync() on all Append() calls
I20260807 07:06:29.647621 15797 log.cc:824] T 0b13db9070bd4d548ddd0fb8543bd421 P d4e4ede98de94f9d893305084c3926dc: Log is configured to *not* fsync() on all Append() calls
I20260807 07:06:29.650866 15797 tablet_bootstrap.cc:492] T 0b13db9070bd4d548ddd0fb8543bd421 P d4e4ede98de94f9d893305084c3926dc: No bootstrap required, opened a new log
I20260807 07:06:29.651156 15797 ts_tablet_manager.cc:1397] T 0b13db9070bd4d548ddd0fb8543bd421 P d4e4ede98de94f9d893305084c3926dc: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260807 07:06:29.653304 15796 tablet_bootstrap.cc:492] T 0b13db9070bd4d548ddd0fb8543bd421 P 48e2225ba70a41389cd3b752f839fb62: No bootstrap required, opened a new log
I20260807 07:06:29.653622 15796 ts_tablet_manager.cc:1397] T 0b13db9070bd4d548ddd0fb8543bd421 P 48e2225ba70a41389cd3b752f839fb62: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.006s
I20260807 07:06:29.653692 15793 raft_consensus.cc:348] T 0b13db9070bd4d548ddd0fb8543bd421 P 72406d752c154c49a65f7f1ccdb95e57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e2225ba70a41389cd3b752f839fb62" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 33293 } } peers { permanent_uuid: "d4e4ede98de94f9d893305084c3926dc" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 34393 } } peers { permanent_uuid: "72406d752c154c49a65f7f1ccdb95e57" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 33087 } }
I20260807 07:06:29.654211 15793 raft_consensus.cc:374] T 0b13db9070bd4d548ddd0fb8543bd421 P 72406d752c154c49a65f7f1ccdb95e57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:06:29.654328 15793 raft_consensus.cc:729] T 0b13db9070bd4d548ddd0fb8543bd421 P 72406d752c154c49a65f7f1ccdb95e57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72406d752c154c49a65f7f1ccdb95e57, State: Initialized, Role: FOLLOWER
I20260807 07:06:29.655017 15793 consensus_queue.cc:260] T 0b13db9070bd4d548ddd0fb8543bd421 P 72406d752c154c49a65f7f1ccdb95e57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e2225ba70a41389cd3b752f839fb62" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 33293 } } peers { permanent_uuid: "d4e4ede98de94f9d893305084c3926dc" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 34393 } } peers { permanent_uuid: "72406d752c154c49a65f7f1ccdb95e57" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 33087 } }
I20260807 07:06:29.656286 15765 heartbeater.cc:503] Master 127.21.83.62:39941 was elected leader, sending a full tablet report...
I20260807 07:06:29.657002 15793 ts_tablet_manager.cc:1428] T 0b13db9070bd4d548ddd0fb8543bd421 P 72406d752c154c49a65f7f1ccdb95e57: Time spent starting tablet: real 0.012s user 0.014s sys 0.000s
I20260807 07:06:29.661283 15797 raft_consensus.cc:348] T 0b13db9070bd4d548ddd0fb8543bd421 P d4e4ede98de94f9d893305084c3926dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e2225ba70a41389cd3b752f839fb62" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 33293 } } peers { permanent_uuid: "d4e4ede98de94f9d893305084c3926dc" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 34393 } } peers { permanent_uuid: "72406d752c154c49a65f7f1ccdb95e57" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 33087 } }
I20260807 07:06:29.661729 15797 raft_consensus.cc:374] T 0b13db9070bd4d548ddd0fb8543bd421 P d4e4ede98de94f9d893305084c3926dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:06:29.661845 15797 raft_consensus.cc:729] T 0b13db9070bd4d548ddd0fb8543bd421 P d4e4ede98de94f9d893305084c3926dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4e4ede98de94f9d893305084c3926dc, State: Initialized, Role: FOLLOWER
I20260807 07:06:29.662571 15797 consensus_queue.cc:260] T 0b13db9070bd4d548ddd0fb8543bd421 P d4e4ede98de94f9d893305084c3926dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e2225ba70a41389cd3b752f839fb62" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 33293 } } peers { permanent_uuid: "d4e4ede98de94f9d893305084c3926dc" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 34393 } } peers { permanent_uuid: "72406d752c154c49a65f7f1ccdb95e57" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 33087 } }
I20260807 07:06:29.663033 15796 raft_consensus.cc:348] T 0b13db9070bd4d548ddd0fb8543bd421 P 48e2225ba70a41389cd3b752f839fb62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e2225ba70a41389cd3b752f839fb62" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 33293 } } peers { permanent_uuid: "d4e4ede98de94f9d893305084c3926dc" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 34393 } } peers { permanent_uuid: "72406d752c154c49a65f7f1ccdb95e57" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 33087 } }
I20260807 07:06:29.663462 15796 raft_consensus.cc:374] T 0b13db9070bd4d548ddd0fb8543bd421 P 48e2225ba70a41389cd3b752f839fb62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:06:29.663568 15796 raft_consensus.cc:729] T 0b13db9070bd4d548ddd0fb8543bd421 P 48e2225ba70a41389cd3b752f839fb62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e2225ba70a41389cd3b752f839fb62, State: Initialized, Role: FOLLOWER
I20260807 07:06:29.664111 15796 consensus_queue.cc:260] T 0b13db9070bd4d548ddd0fb8543bd421 P 48e2225ba70a41389cd3b752f839fb62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e2225ba70a41389cd3b752f839fb62" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 33293 } } peers { permanent_uuid: "d4e4ede98de94f9d893305084c3926dc" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 34393 } } peers { permanent_uuid: "72406d752c154c49a65f7f1ccdb95e57" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 33087 } }
I20260807 07:06:29.664541 15797 ts_tablet_manager.cc:1428] T 0b13db9070bd4d548ddd0fb8543bd421 P d4e4ede98de94f9d893305084c3926dc: Time spent starting tablet: real 0.013s user 0.006s sys 0.006s
I20260807 07:06:29.665441 15585 heartbeater.cc:503] Master 127.21.83.62:39941 was elected leader, sending a full tablet report...
I20260807 07:06:29.665962 15796 ts_tablet_manager.cc:1428] T 0b13db9070bd4d548ddd0fb8543bd421 P 48e2225ba70a41389cd3b752f839fb62: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260807 07:06:29.750608 15803 raft_consensus.cc:482] T 0b13db9070bd4d548ddd0fb8543bd421 P 72406d752c154c49a65f7f1ccdb95e57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:06:29.750919 15803 raft_consensus.cc:504] T 0b13db9070bd4d548ddd0fb8543bd421 P 72406d752c154c49a65f7f1ccdb95e57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e2225ba70a41389cd3b752f839fb62" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 33293 } } peers { permanent_uuid: "d4e4ede98de94f9d893305084c3926dc" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 34393 } } peers { permanent_uuid: "72406d752c154c49a65f7f1ccdb95e57" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 33087 } }
I20260807 07:06:29.752586 15803 leader_election.cc:288] T 0b13db9070bd4d548ddd0fb8543bd421 P 72406d752c154c49a65f7f1ccdb95e57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48e2225ba70a41389cd3b752f839fb62 (127.21.83.2:33293), d4e4ede98de94f9d893305084c3926dc (127.21.83.1:34393)
I20260807 07:06:29.758379 15784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:29.753120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:33937 (local address 127.21.83.2:33293)
0807 07:06:29.753308 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:29.753314 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:29.753357 (+ 43us) server_negotiation.cc:407] Connection header received
0807 07:06:29.753433 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:29.753439 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:29.753517 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:29.753589 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:06:29.754143 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:29.755120 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:29.756744 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:29.757541 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:29.757656 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:29.757745 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:29.758002 (+ 257us) server_negotiation.cc:299] Negotiation successful
0807 07:06:29.758122 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260807 07:06:29.758385 15767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:29.752920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.2:33293 (local address 127.21.83.3:33937)
0807 07:06:29.753115 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:29.753134 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 07:06:29.753259 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:06:29.753665 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:06:29.753685 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:06:29.753720 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:06:29.753966 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:29.753979 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:29.755237 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:29.755241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:29.756596 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:29.756609 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:29.757689 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:29.757695 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:29.757838 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:29.757863 (+ 25us) client_negotiation.cc:769] Sending connection context
0807 07:06:29.757952 (+ 89us) client_negotiation.cc:240] Negotiation successful
0807 07:06:29.758084 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260807 07:06:29.759050 15806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:29.753405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:34393 (local address 127.21.83.3:36797)
0807 07:06:29.754095 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:29.754112 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 07:06:29.754242 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:06:29.754548 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:06:29.754553 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:06:29.754577 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:06:29.754901 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:29.754911 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:29.755835 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:29.755841 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:29.757218 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:29.757230 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:29.758342 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:29.758348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:29.758529 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:29.758565 (+ 36us) client_negotiation.cc:769] Sending connection context
0807 07:06:29.758647 (+ 82us) client_negotiation.cc:240] Negotiation successful
0807 07:06:29.758765 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":366,"threads_started":1}
I20260807 07:06:29.759045 15789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:29.753187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:36797 (local address 127.21.83.1:34393)
0807 07:06:29.753402 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:29.753407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:29.754198 (+ 791us) server_negotiation.cc:407] Connection header received
0807 07:06:29.754349 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:29.754353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:29.754439 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:29.754508 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:06:29.755041 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:29.755683 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:29.757396 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:29.758177 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:29.758288 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:29.758377 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:29.758700 (+ 323us) server_negotiation.cc:299] Negotiation successful
0807 07:06:29.758829 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 07:06:29.759416 15521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b13db9070bd4d548ddd0fb8543bd421" candidate_uuid: "72406d752c154c49a65f7f1ccdb95e57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e2225ba70a41389cd3b752f839fb62" is_pre_election: true
I20260807 07:06:29.759923 15521 raft_consensus.cc:2454] T 0b13db9070bd4d548ddd0fb8543bd421 P 48e2225ba70a41389cd3b752f839fb62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72406d752c154c49a65f7f1ccdb95e57 in term 0.
I20260807 07:06:29.759943 15322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b13db9070bd4d548ddd0fb8543bd421" candidate_uuid: "72406d752c154c49a65f7f1ccdb95e57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4e4ede98de94f9d893305084c3926dc" is_pre_election: true
I20260807 07:06:29.760494 15322 raft_consensus.cc:2454] T 0b13db9070bd4d548ddd0fb8543bd421 P d4e4ede98de94f9d893305084c3926dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72406d752c154c49a65f7f1ccdb95e57 in term 0.
I20260807 07:06:29.760614 15616 leader_election.cc:302] T 0b13db9070bd4d548ddd0fb8543bd421 P 72406d752c154c49a65f7f1ccdb95e57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e2225ba70a41389cd3b752f839fb62, 72406d752c154c49a65f7f1ccdb95e57; no voters:
I20260807 07:06:29.761090 15803 raft_consensus.cc:2781] T 0b13db9070bd4d548ddd0fb8543bd421 P 72406d752c154c49a65f7f1ccdb95e57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 07:06:29.761251 15803 raft_consensus.cc:482] T 0b13db9070bd4d548ddd0fb8543bd421 P 72406d752c154c49a65f7f1ccdb95e57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:06:29.761361 15803 raft_consensus.cc:3037] T 0b13db9070bd4d548ddd0fb8543bd421 P 72406d752c154c49a65f7f1ccdb95e57 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:06:29.763522 15803 raft_consensus.cc:504] T 0b13db9070bd4d548ddd0fb8543bd421 P 72406d752c154c49a65f7f1ccdb95e57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e2225ba70a41389cd3b752f839fb62" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 33293 } } peers { permanent_uuid: "d4e4ede98de94f9d893305084c3926dc" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 34393 } } peers { permanent_uuid: "72406d752c154c49a65f7f1ccdb95e57" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 33087 } }
I20260807 07:06:29.764067 15803 leader_election.cc:288] T 0b13db9070bd4d548ddd0fb8543bd421 P 72406d752c154c49a65f7f1ccdb95e57 [CANDIDATE]: Term 1 election: Requested vote from peers 48e2225ba70a41389cd3b752f839fb62 (127.21.83.2:33293), d4e4ede98de94f9d893305084c3926dc (127.21.83.1:34393)
I20260807 07:06:29.764454 15521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b13db9070bd4d548ddd0fb8543bd421" candidate_uuid: "72406d752c154c49a65f7f1ccdb95e57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e2225ba70a41389cd3b752f839fb62"
I20260807 07:06:29.764488 15322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b13db9070bd4d548ddd0fb8543bd421" candidate_uuid: "72406d752c154c49a65f7f1ccdb95e57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4e4ede98de94f9d893305084c3926dc"
I20260807 07:06:29.764640 15521 raft_consensus.cc:3037] T 0b13db9070bd4d548ddd0fb8543bd421 P 48e2225ba70a41389cd3b752f839fb62 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:06:29.764652 15322 raft_consensus.cc:3037] T 0b13db9070bd4d548ddd0fb8543bd421 P d4e4ede98de94f9d893305084c3926dc [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:06:29.766875 15322 raft_consensus.cc:2454] T 0b13db9070bd4d548ddd0fb8543bd421 P d4e4ede98de94f9d893305084c3926dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72406d752c154c49a65f7f1ccdb95e57 in term 1.
I20260807 07:06:29.766896 15521 raft_consensus.cc:2454] T 0b13db9070bd4d548ddd0fb8543bd421 P 48e2225ba70a41389cd3b752f839fb62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72406d752c154c49a65f7f1ccdb95e57 in term 1.
I20260807 07:06:29.767251 15617 leader_election.cc:302] T 0b13db9070bd4d548ddd0fb8543bd421 P 72406d752c154c49a65f7f1ccdb95e57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72406d752c154c49a65f7f1ccdb95e57, d4e4ede98de94f9d893305084c3926dc; no voters:
I20260807 07:06:29.767573 15803 raft_consensus.cc:2781] T 0b13db9070bd4d548ddd0fb8543bd421 P 72406d752c154c49a65f7f1ccdb95e57 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:06:29.768049 15803 raft_consensus.cc:686] T 0b13db9070bd4d548ddd0fb8543bd421 P 72406d752c154c49a65f7f1ccdb95e57 [term 1 LEADER]: Becoming Leader. State: Replica: 72406d752c154c49a65f7f1ccdb95e57, State: Running, Role: LEADER
I20260807 07:06:29.768522 15803 consensus_queue.cc:237] T 0b13db9070bd4d548ddd0fb8543bd421 P 72406d752c154c49a65f7f1ccdb95e57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e2225ba70a41389cd3b752f839fb62" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 33293 } } peers { permanent_uuid: "d4e4ede98de94f9d893305084c3926dc" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 34393 } } peers { permanent_uuid: "72406d752c154c49a65f7f1ccdb95e57" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 33087 } }
I20260807 07:06:29.772953 15115 catalog_manager.cc:5495] T 0b13db9070bd4d548ddd0fb8543bd421 P 72406d752c154c49a65f7f1ccdb95e57 reported cstate change: term changed from 0 to 1, leader changed from <none> to 72406d752c154c49a65f7f1ccdb95e57 (127.21.83.3). New cstate: current_term: 1 leader_uuid: "72406d752c154c49a65f7f1ccdb95e57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e2225ba70a41389cd3b752f839fb62" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 33293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4e4ede98de94f9d893305084c3926dc" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 34393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72406d752c154c49a65f7f1ccdb95e57" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 33087 } health_report { overall_health: HEALTHY } } }
W20260807 07:06:29.801294 15766 tablet.cc:2369] T 0b13db9070bd4d548ddd0fb8543bd421 P 72406d752c154c49a65f7f1ccdb95e57: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 07:06:29.809119 15791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:29.804001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37052 (local address 127.21.83.3:33087)
0807 07:06:29.804202 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:29.804207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:29.804226 (+ 19us) server_negotiation.cc:407] Connection header received
0807 07:06:29.804387 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:29.804391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:29.804465 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:29.804587 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:06:29.804982 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:29.805719 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:29.807090 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:29.807497 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:29.807604 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:29.808082 (+ 478us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:06:29.808546 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:06:29.808684 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:29.808797 (+ 113us) server_negotiation.cc:299] Negotiation successful
0807 07:06:29.808874 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260807 07:06:29.813503 15587 tablet.cc:2369] T 0b13db9070bd4d548ddd0fb8543bd421 P 48e2225ba70a41389cd3b752f839fb62: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 07:06:29.816779 15521 raft_consensus.cc:1261] T 0b13db9070bd4d548ddd0fb8543bd421 P 48e2225ba70a41389cd3b752f839fb62 [term 1 FOLLOWER]: Refusing update from remote peer 72406d752c154c49a65f7f1ccdb95e57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:06:29.816794 15322 raft_consensus.cc:1261] T 0b13db9070bd4d548ddd0fb8543bd421 P d4e4ede98de94f9d893305084c3926dc [term 1 FOLLOWER]: Refusing update from remote peer 72406d752c154c49a65f7f1ccdb95e57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:06:29.817670 15807 consensus_queue.cc:1035] T 0b13db9070bd4d548ddd0fb8543bd421 P 72406d752c154c49a65f7f1ccdb95e57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e2225ba70a41389cd3b752f839fb62" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:06:29.818011 15803 consensus_queue.cc:1035] T 0b13db9070bd4d548ddd0fb8543bd421 P 72406d752c154c49a65f7f1ccdb95e57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4e4ede98de94f9d893305084c3926dc" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 34393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:06:29.829110 15815 mvcc.cc:205] Tried to move back new op lower bound from 7315809852680597504 to 7315809852495785984. Current Snapshot: MvccSnapshot[applied={T|T < 7315809852680597504}]
I20260807 07:06:29.841945 15115 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:06:29.842260 15115 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:06:29.845784 15115 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b57daf68acbc449b916ec0d167b3c6a7: Sending DeleteTablet for 3 replicas of tablet 0b13db9070bd4d548ddd0fb8543bd421
I20260807 07:06:29.846817 15292 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b13db9070bd4d548ddd0fb8543bd421 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:06:29 UTC) from {username='slave'} at 127.0.0.1:58812
I20260807 07:06:29.847069 15489 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b13db9070bd4d548ddd0fb8543bd421 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:06:29 UTC) from {username='slave'} at 127.0.0.1:33114
I20260807 07:06:29.847221 15677 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b13db9070bd4d548ddd0fb8543bd421 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:06:29 UTC) from {username='slave'} at 127.0.0.1:37036
I20260807 07:06:29.848702 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 15198
I20260807 07:06:29.856604 15829 tablet_replica.cc:330] T 0b13db9070bd4d548ddd0fb8543bd421 P 48e2225ba70a41389cd3b752f839fb62: stopping tablet replica
I20260807 07:06:29.857062 15829 raft_consensus.cc:2229] T 0b13db9070bd4d548ddd0fb8543bd421 P 48e2225ba70a41389cd3b752f839fb62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:06:29.857561 15829 raft_consensus.cc:2258] T 0b13db9070bd4d548ddd0fb8543bd421 P 48e2225ba70a41389cd3b752f839fb62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:06:29.860790 15830 tablet_replica.cc:330] T 0b13db9070bd4d548ddd0fb8543bd421 P 72406d752c154c49a65f7f1ccdb95e57: stopping tablet replica
I20260807 07:06:29.861212 15830 raft_consensus.cc:2229] T 0b13db9070bd4d548ddd0fb8543bd421 P 72406d752c154c49a65f7f1ccdb95e57 [term 1 LEADER]: Raft consensus shutting down.
I20260807 07:06:29.861905 15830 raft_consensus.cc:2258] T 0b13db9070bd4d548ddd0fb8543bd421 P 72406d752c154c49a65f7f1ccdb95e57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:06:29.865083 15830 ts_tablet_manager.cc:1905] T 0b13db9070bd4d548ddd0fb8543bd421 P 72406d752c154c49a65f7f1ccdb95e57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:06:29.870313 15829 ts_tablet_manager.cc:1905] T 0b13db9070bd4d548ddd0fb8543bd421 P 48e2225ba70a41389cd3b752f839fb62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:06:29.874040 15830 ts_tablet_manager.cc:1918] T 0b13db9070bd4d548ddd0fb8543bd421 P 72406d752c154c49a65f7f1ccdb95e57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:06:29.874454 15830 log.cc:1196] T 0b13db9070bd4d548ddd0fb8543bd421 P 72406d752c154c49a65f7f1ccdb95e57: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-2/wal/wals/0b13db9070bd4d548ddd0fb8543bd421
I20260807 07:06:29.875166 15830 ts_tablet_manager.cc:1939] T 0b13db9070bd4d548ddd0fb8543bd421 P 72406d752c154c49a65f7f1ccdb95e57: Deleting consensus metadata
I20260807 07:06:29.877508 15094 catalog_manager.cc:4841] TS 72406d752c154c49a65f7f1ccdb95e57 (127.21.83.3:33087): tablet 0b13db9070bd4d548ddd0fb8543bd421 (table test-table [id=1c108a2163e44224a3e0ade2949f8b7f]) successfully deleted
I20260807 07:06:29.880467 15828 tablet_replica.cc:330] T 0b13db9070bd4d548ddd0fb8543bd421 P d4e4ede98de94f9d893305084c3926dc: stopping tablet replica
I20260807 07:06:29.880935 15828 raft_consensus.cc:2229] T 0b13db9070bd4d548ddd0fb8543bd421 P d4e4ede98de94f9d893305084c3926dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:06:29.881430 15828 raft_consensus.cc:2258] T 0b13db9070bd4d548ddd0fb8543bd421 P d4e4ede98de94f9d893305084c3926dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:06:29.882128 15829 ts_tablet_manager.cc:1918] T 0b13db9070bd4d548ddd0fb8543bd421 P 48e2225ba70a41389cd3b752f839fb62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:06:29.882383 15829 log.cc:1196] T 0b13db9070bd4d548ddd0fb8543bd421 P 48e2225ba70a41389cd3b752f839fb62: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786086078080656-21836-0/minicluster-data/ts-1/wal/wals/0b13db9070bd4d548ddd0fb8543bd421
I20260807 07:06:29.882968 15829 ts_tablet_manager.cc:1939] T 0b13db9070bd4d548ddd0fb8543bd421 P 48e2225ba70a41389cd3b752f839fb62: Deleting consensus metadata
I20260807 07:06:29.884193 15789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:29.850080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58826 (local address 127.21.83.1:34393)
0807 07:06:29.850273 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:29.850279 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:29.862653 (+ 12374us) server_negotiation.cc:407] Connection header received
0807 07:06:29.863369 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:29.863377 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:29.863520 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:29.863599 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:29.864387 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:29.865750 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:29.877393 (+ 11643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:29.878560 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:29.878734 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:29.882251 (+ 3517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:29.882266 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:29.882283 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:06:29.883086 (+ 803us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:29.883158 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:06:29.883203 (+ 45us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:29.883326 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:29.883675 (+ 349us) server_negotiation.cc:299] Negotiation successful
0807 07:06:29.883823 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 07:06:29.884222 15828 ts_tablet_manager.cc:1905] T 0b13db9070bd4d548ddd0fb8543bd421 P d4e4ede98de94f9d893305084c3926dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:06:29.885692 15385 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:06:29.885670 15094 catalog_manager.cc:4841] TS 48e2225ba70a41389cd3b752f839fb62 (127.21.83.2:33293): tablet 0b13db9070bd4d548ddd0fb8543bd421 (table test-table [id=1c108a2163e44224a3e0ade2949f8b7f]) successfully deleted
I20260807 07:06:29.981045 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 15198
I20260807 07:06:30.003274 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 15396
W20260807 07:06:30.003291 15096 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.83.1:34393 (error 108)
I20260807 07:06:30.005666 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:30.005185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:34393 (local address 127.0.0.1:58832)
0807 07:06:30.005351 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:30.005480 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.1:34393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260807 07:06:30.005975 15096 catalog_manager.cc:4584] TS d4e4ede98de94f9d893305084c3926dc (127.21.83.1:34393): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0b13db9070bd4d548ddd0fb8543bd421: Network error: Client connection negotiation failed: client connection to 127.21.83.1:34393: connect: Connection refused (error 111)
I20260807 07:06:30.012084 15784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:30.004636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33122 (local address 127.21.83.2:33293)
0807 07:06:30.004827 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:30.004833 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:30.005537 (+ 704us) server_negotiation.cc:407] Connection header received
0807 07:06:30.006375 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:30.006381 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:30.006477 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:30.006567 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:30.006969 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:30.007738 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:30.009255 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:30.009677 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:30.009788 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:30.010777 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:30.010786 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:30.010799 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:06:30.011339 (+ 540us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:30.011385 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:06:30.011416 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:30.011513 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:30.011646 (+ 133us) server_negotiation.cc:299] Negotiation successful
0807 07:06:30.011766 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 07:06:30.012774 15580 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:06:30.059656 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:30.059092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:34393 (local address 127.0.0.1:58836)
0807 07:06:30.059340 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:30.059457 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.1:34393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260807 07:06:30.060842 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:30.060430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:34393 (local address 127.0.0.1:58840)
0807 07:06:30.060583 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:30.060682 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.1:34393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260807 07:06:30.109633 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 15396
I20260807 07:06:30.121299 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:30.120849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:34393 (local address 127.0.0.1:58850)
0807 07:06:30.121052 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:30.121140 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.1:34393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260807 07:06:30.122272 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:30.121934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:34393 (local address 127.0.0.1:58864)
0807 07:06:30.122077 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:30.122150 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.1:34393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260807 07:06:30.141388 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 15591
I20260807 07:06:30.148507 15791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:30.142217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37064 (local address 127.21.83.3:33087)
0807 07:06:30.142428 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:30.142433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:30.142455 (+ 22us) server_negotiation.cc:407] Connection header received
0807 07:06:30.142570 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:30.142575 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:30.142686 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:30.142781 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:30.143190 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:30.143993 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:30.145373 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:30.145837 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:30.145954 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:30.147053 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:30.147066 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:30.147081 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:06:30.147708 (+ 627us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:30.147761 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:06:30.147781 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:30.147878 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:30.148024 (+ 146us) server_negotiation.cc:299] Negotiation successful
0807 07:06:30.148127 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 07:06:30.149256 15759 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:06:30.202963 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:30.202424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:34393 (local address 127.0.0.1:58880)
0807 07:06:30.202682 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:30.202780 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.1:34393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260807 07:06:30.204046 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:30.203640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:34393 (local address 127.0.0.1:58896)
0807 07:06:30.203856 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:30.203927 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.1:34393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260807 07:06:30.273358 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 15591
I20260807 07:06:30.296525 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 15066
I20260807 07:06:30.297591 15159 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:06:30.408175 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 15066
I20260807 07:06:30.408255 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:30.407646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:34393 (local address 127.0.0.1:58902)
0807 07:06:30.407923 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:30.408031 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.1:34393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
2026-08-07T07:06:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2962 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 07:06:30 dist-test-slave-ncp9 krb5kdc[15850](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 07:06:30 dist-test-slave-ncp9 krb5kdc[15850](info): set up 2 sockets
Aug 07 07:06:30 dist-test-slave-ncp9 krb5kdc[15850](info): commencing operation
krb5kdc: starting...
W20260807 07:06:31.656483 21836 mini_kdc.cc:121] Time spent starting KDC: real 1.200s user 0.003s sys 0.055s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Aug 07 07:06:31 dist-test-slave-ncp9 krb5kdc[15850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T07:06:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:06:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
I20260807 07:06:32.005760 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:37755
--webserver_interface=127.21.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:32857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:37755
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:06:32.348225 15869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:06:32.348562 15869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:06:32.348703 15869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:06:32.356244 15869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:06:32.356357 15869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:06:32.356432 15869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:06:32.356510 15869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:06:32.356570 15869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:06:32.366492 15869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:32857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:37755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:37755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:06:32.367848 15869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:06:32.369148 15869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:06:32.380165 15877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:32.380373 15876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:32.380203 15880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:06:32.381174 15869 server_base.cc:1034] running on GCE node
I20260807 07:06:32.381872 15869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:06:32.383222 15869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:06:32.384431 15869 hybrid_clock.cc:648] HybridClock initialized: now 1786086392384385 us; error 32 us; skew 500 ppm
I20260807 07:06:32.384754 15869 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:06:32 dist-test-slave-ncp9 krb5kdc[15850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086392, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:06:32.387835 15869 init.cc:377] Logged in from keytab as kudu/127.21.83.62@KRBTEST.COM (short username kudu)
I20260807 07:06:32.389297 15869 webserver.cc:460] Webserver started at http://127.21.83.62:37171/ using document root <none> and password file <none>
I20260807 07:06:32.389954 15869 fs_manager.cc:359] Metadata directory not provided
I20260807 07:06:32.390074 15869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:06:32.390373 15869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:06:32.393486 15869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "dc25ec5f98a54be883a70158265dddd5"
format_stamp: "Formatted at 2026-08-07 07:06:32 on dist-test-slave-ncp9"
server_key: "2a032fd9913350c5ed081deb9bc5aa46"
server_key_iv: "4e771834b8e49fd675a998d5ddac173a"
server_key_version: "encryptionkey@0"
I20260807 07:06:32.394318 15869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "dc25ec5f98a54be883a70158265dddd5"
format_stamp: "Formatted at 2026-08-07 07:06:32 on dist-test-slave-ncp9"
server_key: "2a032fd9913350c5ed081deb9bc5aa46"
server_key_iv: "4e771834b8e49fd675a998d5ddac173a"
server_key_version: "encryptionkey@0"
I20260807 07:06:32.399858 15869 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 07:06:32.403563 15890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:32.404973 15869 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 07:06:32.405153 15869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "dc25ec5f98a54be883a70158265dddd5"
format_stamp: "Formatted at 2026-08-07 07:06:32 on dist-test-slave-ncp9"
server_key: "2a032fd9913350c5ed081deb9bc5aa46"
server_key_iv: "4e771834b8e49fd675a998d5ddac173a"
server_key_version: "encryptionkey@0"
I20260807 07:06:32.405347 15869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:06:32.432173 15869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:06:32.436420 15869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:06:32.436697 15869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:06:32.455173 15869 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.62:37755
I20260807 07:06:32.455262 15964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.62:37755 every 8 connection(s)
I20260807 07:06:32.456457 15869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:06:32.460031 15965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:06:32.460857 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 15869
I20260807 07:06:32.461084 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:06:32.461700 21836 external_mini_cluster.cc:1428] Setting key 002905f3bb197aefc72237c1b1ef806c
I20260807 07:06:32.473963 15965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc25ec5f98a54be883a70158265dddd5: Bootstrap starting.
Aug 07 07:06:32 dist-test-slave-ncp9 krb5kdc[15850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:06:32.477769 15965 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc25ec5f98a54be883a70158265dddd5: Neither blocks nor log segments found. Creating new log.
I20260807 07:06:32.478861 15965 log.cc:824] T 00000000000000000000000000000000 P dc25ec5f98a54be883a70158265dddd5: Log is configured to *not* fsync() on all Append() calls
I20260807 07:06:32.482215 15965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc25ec5f98a54be883a70158265dddd5: No bootstrap required, opened a new log
I20260807 07:06:32.485332 15968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:32.463516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60512 (local address 127.21.83.62:37755)
0807 07:06:32.464294 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:32.464305 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:32.464349 (+ 44us) server_negotiation.cc:407] Connection header received
0807 07:06:32.466135 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:32.466158 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:32.466512 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:32.466834 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:32.468717 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:32.470059 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:32.471952 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:32.472575 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:32.472725 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:32.477931 (+ 5206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:32.477941 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:32.477955 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:06:32.477997 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:32.481203 (+ 3206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:32.482074 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:32.482080 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:32.482088 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:32.482261 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:32.482633 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:32.482637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:32.482639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:32.483125 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:32.483411 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:32.483906 (+ 495us) server_negotiation.cc:299] Negotiation successful
0807 07:06:32.484300 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":370,"threads_started":1}
I20260807 07:06:32.490541 15965 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc25ec5f98a54be883a70158265dddd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc25ec5f98a54be883a70158265dddd5" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 37755 } }
I20260807 07:06:32.490880 15965 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc25ec5f98a54be883a70158265dddd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:06:32.490976 15965 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc25ec5f98a54be883a70158265dddd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc25ec5f98a54be883a70158265dddd5, State: Initialized, Role: FOLLOWER
I20260807 07:06:32.491505 15965 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc25ec5f98a54be883a70158265dddd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc25ec5f98a54be883a70158265dddd5" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 37755 } }
I20260807 07:06:32.491732 15965 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc25ec5f98a54be883a70158265dddd5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:06:32.491861 15965 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc25ec5f98a54be883a70158265dddd5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:06:32.492004 15965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc25ec5f98a54be883a70158265dddd5 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:06:32.494591 15965 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc25ec5f98a54be883a70158265dddd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc25ec5f98a54be883a70158265dddd5" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 37755 } }
I20260807 07:06:32.495085 15965 leader_election.cc:302] T 00000000000000000000000000000000 P dc25ec5f98a54be883a70158265dddd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc25ec5f98a54be883a70158265dddd5; no voters:
I20260807 07:06:32.495656 15965 leader_election.cc:288] T 00000000000000000000000000000000 P dc25ec5f98a54be883a70158265dddd5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 07:06:32.495779 15970 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc25ec5f98a54be883a70158265dddd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:06:32.496398 15970 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc25ec5f98a54be883a70158265dddd5 [term 1 LEADER]: Becoming Leader. State: Replica: dc25ec5f98a54be883a70158265dddd5, State: Running, Role: LEADER
I20260807 07:06:32.496973 15970 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc25ec5f98a54be883a70158265dddd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc25ec5f98a54be883a70158265dddd5" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 37755 } }
I20260807 07:06:32.497617 15965 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc25ec5f98a54be883a70158265dddd5 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:06:32.500372 15971 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc25ec5f98a54be883a70158265dddd5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc25ec5f98a54be883a70158265dddd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc25ec5f98a54be883a70158265dddd5" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 37755 } } }
I20260807 07:06:32.500458 15972 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc25ec5f98a54be883a70158265dddd5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc25ec5f98a54be883a70158265dddd5. Latest consensus state: current_term: 1 leader_uuid: "dc25ec5f98a54be883a70158265dddd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc25ec5f98a54be883a70158265dddd5" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 37755 } } }
I20260807 07:06:32.500761 15971 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc25ec5f98a54be883a70158265dddd5 [sys.catalog]: This master's current role is: LEADER
I20260807 07:06:32.500809 15972 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc25ec5f98a54be883a70158265dddd5 [sys.catalog]: This master's current role is: LEADER
I20260807 07:06:32.502257 15981 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:06:32.506793 15981 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:06:32.514463 15981 catalog_manager.cc:1347] Generated new cluster ID: b9cd944aac6c4b4db6dd6538ce7c3319
I20260807 07:06:32.514562 15981 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:06:32.533033 15981 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:06:32.535038 15981 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:06:32.553503 15981 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc25ec5f98a54be883a70158265dddd5: Generated new TSK 0
I20260807 07:06:32.554631 15981 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.83.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab.
Entry for principal kudu/127.21.83.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab.
I20260807 07:06:32.642100 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.1:0
--local_ip_for_outbound_sockets=127.21.83.1
--webserver_interface=127.21.83.1
--webserver_port=0
--tserver_master_addrs=127.21.83.62:37755
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab
--principal=kudu/127.21.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:32857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:06:32.984856 16000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:06:32.985155 16000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:06:32.985304 16000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:06:32.992630 16000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:06:32.992761 16000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:06:32.992956 16000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.1
I20260807 07:06:33.003926 16000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:32857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab
--principal=kudu/127.21.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.83.1
--webserver_port=0
--tserver_master_addrs=127.21.83.62:37755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.1
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:06:33.005355 16000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:06:33.006690 16000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:06:33.020397 16007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:33.020406 16008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:33.020421 16013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:06:33.021230 16000 server_base.cc:1034] running on GCE node
I20260807 07:06:33.021939 16000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:06:33.023000 16000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:06:33.024263 16000 hybrid_clock.cc:648] HybridClock initialized: now 1786086393024194 us; error 58 us; skew 500 ppm
I20260807 07:06:33.024652 16000 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:06:33 dist-test-slave-ncp9 krb5kdc[15850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086393, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:06:33.032531 16000 init.cc:377] Logged in from keytab as kudu/127.21.83.1@KRBTEST.COM (short username kudu)
I20260807 07:06:33.034436 16000 webserver.cc:460] Webserver started at http://127.21.83.1:39325/ using document root <none> and password file <none>
I20260807 07:06:33.035225 16000 fs_manager.cc:359] Metadata directory not provided
I20260807 07:06:33.035367 16000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:06:33.035750 16000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:06:33.038909 16000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-0/data/instance:
uuid: "53f31bf11eff49e2bf52819dac529388"
format_stamp: "Formatted at 2026-08-07 07:06:33 on dist-test-slave-ncp9"
server_key: "3eb7d77fd205f219905e2226e429611b"
server_key_iv: "6597aba9e06ac82425ea9e9c5aec24ce"
server_key_version: "encryptionkey@0"
I20260807 07:06:33.039728 16000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-0/wal/instance:
uuid: "53f31bf11eff49e2bf52819dac529388"
format_stamp: "Formatted at 2026-08-07 07:06:33 on dist-test-slave-ncp9"
server_key: "3eb7d77fd205f219905e2226e429611b"
server_key_iv: "6597aba9e06ac82425ea9e9c5aec24ce"
server_key_version: "encryptionkey@0"
I20260807 07:06:33.045038 16000 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260807 07:06:33.048388 16026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:33.049664 16000 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 07:06:33.049815 16000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-0/wal
uuid: "53f31bf11eff49e2bf52819dac529388"
format_stamp: "Formatted at 2026-08-07 07:06:33 on dist-test-slave-ncp9"
server_key: "3eb7d77fd205f219905e2226e429611b"
server_key_iv: "6597aba9e06ac82425ea9e9c5aec24ce"
server_key_version: "encryptionkey@0"
I20260807 07:06:33.049993 16000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:06:33.069192 16000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:06:33.072727 16000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:06:33.073026 16000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:06:33.073920 16000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:06:33.075204 16000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:06:33.075276 16000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:33.075393 16000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:06:33.075443 16000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:33.106606 16000 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.1:41813
I20260807 07:06:33.106703 16198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.1:41813 every 8 connection(s)
I20260807 07:06:33.107867 16000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
I20260807 07:06:33.117244 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 16000
I20260807 07:06:33.117427 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-0/wal/instance
I20260807 07:06:33.117900 21836 external_mini_cluster.cc:1428] Setting key 149dfd55f82fd833ba74080cce034b31
Aug 07 07:06:33 dist-test-slave-ncp9 krb5kdc[15850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086393, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.1@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:06:33.126667 16202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:33.110340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:52199 (local address 127.21.83.62:37755)
0807 07:06:33.110801 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:33.110806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:33.111558 (+ 752us) server_negotiation.cc:407] Connection header received
0807 07:06:33.113353 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:33.113359 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:33.113441 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:33.113543 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:33.114938 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:33.115673 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:33.117293 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:33.117740 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:33.117846 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:33.121299 (+ 3453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:33.121305 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:33.121308 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:06:33.121338 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:33.123742 (+ 2404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:33.124439 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:33.124443 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:33.124446 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:33.124576 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:33.125000 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:33.125004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:33.125006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:33.125260 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:33.125411 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:33.126068 (+ 657us) server_negotiation.cc:299] Negotiation successful
0807 07:06:33.126305 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":253,"threads_started":1}
I20260807 07:06:33.127516 16203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:33.110736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:37755 (local address 127.21.83.1:52199)
0807 07:06:33.111405 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:33.111447 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 07:06:33.113108 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:06:33.113765 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:06:33.113773 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:06:33.114256 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:06:33.114771 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:33.114787 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:33.115828 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:33.115832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:33.117149 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:33.117164 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:33.117969 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:33.117988 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:33.118262 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:33.118914 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:06:33.118941 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:06:33.121098 (+ 2157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:06:33.123870 (+ 2772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:06:33.123878 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:06:33.123891 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:06:33.124316 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:06:33.124676 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:06:33.124681 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:06:33.124686 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:06:33.124872 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:06:33.125384 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:06:33.125392 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:06:33.125696 (+ 304us) client_negotiation.cc:769] Sending connection context
0807 07:06:33.125918 (+ 222us) client_negotiation.cc:240] Negotiation successful
0807 07:06:33.126419 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":254,"threads_started":1}
I20260807 07:06:33.129041 16200 heartbeater.cc:347] Connected to a master server at 127.21.83.62:37755
I20260807 07:06:33.129362 16200 heartbeater.cc:464] Registering TS with master...
I20260807 07:06:33.130206 16200 heartbeater.cc:511] Master 127.21.83.62:37755 requested a full tablet report, sending...
I20260807 07:06:33.132391 15917 ts_manager.cc:194] Registered new tserver with Master: 53f31bf11eff49e2bf52819dac529388 (127.21.83.1:41813)
I20260807 07:06:33.133852 15917 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.1@KRBTEST.COM'} at 127.21.83.1:52199
WARNING: no policy specified for kudu/127.21.83.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab.
Entry for principal kudu/127.21.83.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab.
I20260807 07:06:33.182096 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.2:0
--local_ip_for_outbound_sockets=127.21.83.2
--webserver_interface=127.21.83.2
--webserver_port=0
--tserver_master_addrs=127.21.83.62:37755
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab
--principal=kudu/127.21.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:32857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:06:33.513772 16206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:06:33.514032 16206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:06:33.514161 16206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:06:33.521219 16206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:06:33.521315 16206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:06:33.521461 16206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.2
I20260807 07:06:33.531674 16206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:32857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab
--principal=kudu/127.21.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.83.2
--webserver_port=0
--tserver_master_addrs=127.21.83.62:37755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.2
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:06:33.532851 16206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:06:33.534188 16206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:06:33.549163 16212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:33.549247 16213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:33.549225 16222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:06:33.550273 16206 server_base.cc:1034] running on GCE node
I20260807 07:06:33.550992 16206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:06:33.552086 16206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:06:33.553292 16206 hybrid_clock.cc:648] HybridClock initialized: now 1786086393553221 us; error 63 us; skew 500 ppm
I20260807 07:06:33.553663 16206 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:06:33 dist-test-slave-ncp9 krb5kdc[15850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086393, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:06:33.557024 16206 init.cc:377] Logged in from keytab as kudu/127.21.83.2@KRBTEST.COM (short username kudu)
I20260807 07:06:33.558595 16206 webserver.cc:460] Webserver started at http://127.21.83.2:39403/ using document root <none> and password file <none>
I20260807 07:06:33.559345 16206 fs_manager.cc:359] Metadata directory not provided
I20260807 07:06:33.559469 16206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:06:33.559819 16206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:06:33.563091 16206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-1/data/instance:
uuid: "a43303fd25b849e485c8cc57c9f1fe58"
format_stamp: "Formatted at 2026-08-07 07:06:33 on dist-test-slave-ncp9"
server_key: "83faf089e3ca96714444d4d7b59968ff"
server_key_iv: "7f4d2c9b5d7ab1c775bfe78827005513"
server_key_version: "encryptionkey@0"
I20260807 07:06:33.563897 16206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-1/wal/instance:
uuid: "a43303fd25b849e485c8cc57c9f1fe58"
format_stamp: "Formatted at 2026-08-07 07:06:33 on dist-test-slave-ncp9"
server_key: "83faf089e3ca96714444d4d7b59968ff"
server_key_iv: "7f4d2c9b5d7ab1c775bfe78827005513"
server_key_version: "encryptionkey@0"
I20260807 07:06:33.569756 16206 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 07:06:33.573645 16231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:33.575162 16206 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 07:06:33.575363 16206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-1/wal
uuid: "a43303fd25b849e485c8cc57c9f1fe58"
format_stamp: "Formatted at 2026-08-07 07:06:33 on dist-test-slave-ncp9"
server_key: "83faf089e3ca96714444d4d7b59968ff"
server_key_iv: "7f4d2c9b5d7ab1c775bfe78827005513"
server_key_version: "encryptionkey@0"
I20260807 07:06:33.575582 16206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:06:33.592798 16206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:06:33.597369 16206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:06:33.597657 16206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:06:33.598719 16206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:06:33.600457 16206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:06:33.600540 16206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:33.600662 16206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:06:33.600725 16206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:33.632359 16206 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.2:34335
I20260807 07:06:33.632418 16396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.2:34335 every 8 connection(s)
I20260807 07:06:33.633733 16206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
I20260807 07:06:33.642478 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 16206
I20260807 07:06:33.642688 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-1/wal/instance
I20260807 07:06:33.643195 21836 external_mini_cluster.cc:1428] Setting key a9d0daa3c9e0bc5b6e6efefd9fb342d5
Aug 07 07:06:33 dist-test-slave-ncp9 krb5kdc[15850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086393, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.2@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:06:33.655651 16400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:33.636406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:55985 (local address 127.21.83.62:37755)
0807 07:06:33.637047 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:33.637052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:33.637990 (+ 938us) server_negotiation.cc:407] Connection header received
0807 07:06:33.639955 (+ 1965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:33.639961 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:33.640045 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:33.640144 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:33.641832 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:33.642849 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:33.644667 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:33.645239 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:33.645379 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:33.648959 (+ 3580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:33.648966 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:33.648971 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:06:33.649012 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:33.652095 (+ 3083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:33.653027 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:33.653033 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:33.653036 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:33.653205 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:33.653729 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:33.653734 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:33.653736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:33.654086 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:33.654246 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:33.655036 (+ 790us) server_negotiation.cc:299] Negotiation successful
0807 07:06:33.655271 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":377,"threads_started":1}
I20260807 07:06:33.656632 16402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:33.637068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:37755 (local address 127.21.83.2:55985)
0807 07:06:33.637806 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:33.637854 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 07:06:33.639652 (+ 1798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:06:33.640405 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:06:33.640413 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:06:33.641027 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:06:33.641650 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:33.641669 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:33.643031 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:33.643036 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:33.644513 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:33.644529 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:33.645458 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:33.645464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:33.645689 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:33.646365 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:06:33.646390 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:06:33.648654 (+ 2264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:06:33.652303 (+ 3649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:06:33.652311 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:06:33.652325 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:06:33.652838 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:06:33.653307 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:06:33.653310 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:06:33.653312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:06:33.653607 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:06:33.654216 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:06:33.654223 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:06:33.654658 (+ 435us) client_negotiation.cc:769] Sending connection context
0807 07:06:33.654905 (+ 247us) client_negotiation.cc:240] Negotiation successful
0807 07:06:33.655379 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":274,"threads_started":1}
I20260807 07:06:33.658499 16398 heartbeater.cc:347] Connected to a master server at 127.21.83.62:37755
I20260807 07:06:33.658809 16398 heartbeater.cc:464] Registering TS with master...
I20260807 07:06:33.659614 16398 heartbeater.cc:511] Master 127.21.83.62:37755 requested a full tablet report, sending...
I20260807 07:06:33.661338 15917 ts_manager.cc:194] Registered new tserver with Master: a43303fd25b849e485c8cc57c9f1fe58 (127.21.83.2:34335)
I20260807 07:06:33.662184 15917 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.2@KRBTEST.COM'} at 127.21.83.2:55985
WARNING: no policy specified for kudu/127.21.83.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab.
Entry for principal kudu/127.21.83.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab.
I20260807 07:06:33.715492 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.3:0
--local_ip_for_outbound_sockets=127.21.83.3
--webserver_interface=127.21.83.3
--webserver_port=0
--tserver_master_addrs=127.21.83.62:37755
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab
--principal=kudu/127.21.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:32857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:06:34.058523 16407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:06:34.058804 16407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:06:34.058899 16407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:06:34.066056 16407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:06:34.066165 16407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:06:34.066277 16407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.3
I20260807 07:06:34.075780 16407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:32857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab
--principal=kudu/127.21.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.83.3
--webserver_port=0
--tserver_master_addrs=127.21.83.62:37755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.3
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:06:34.077014 16407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:06:34.078321 16407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:06:34.090596 16415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:34.090672 16413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:34.090641 16412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:06:34.091456 16407 server_base.cc:1034] running on GCE node
I20260807 07:06:34.092031 16407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:06:34.092911 16407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:06:34.094128 16407 hybrid_clock.cc:648] HybridClock initialized: now 1786086394094073 us; error 41 us; skew 500 ppm
I20260807 07:06:34.094425 16407 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:06:34 dist-test-slave-ncp9 krb5kdc[15850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086394, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:06:34.097401 16407 init.cc:377] Logged in from keytab as kudu/127.21.83.3@KRBTEST.COM (short username kudu)
I20260807 07:06:34.098819 16407 webserver.cc:460] Webserver started at http://127.21.83.3:33173/ using document root <none> and password file <none>
I20260807 07:06:34.099463 16407 fs_manager.cc:359] Metadata directory not provided
I20260807 07:06:34.099592 16407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:06:34.099887 16407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:06:34.102815 16407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-2/data/instance:
uuid: "2272bdd8f6544fd4810d853858df7392"
format_stamp: "Formatted at 2026-08-07 07:06:34 on dist-test-slave-ncp9"
server_key: "86de166cb038dd29467632a4c7da0abb"
server_key_iv: "320fdd1b7364b3974d5328d911ef6c73"
server_key_version: "encryptionkey@0"
I20260807 07:06:34.103569 16407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-2/wal/instance:
uuid: "2272bdd8f6544fd4810d853858df7392"
format_stamp: "Formatted at 2026-08-07 07:06:34 on dist-test-slave-ncp9"
server_key: "86de166cb038dd29467632a4c7da0abb"
server_key_iv: "320fdd1b7364b3974d5328d911ef6c73"
server_key_version: "encryptionkey@0"
I20260807 07:06:34.108867 16407 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260807 07:06:34.112344 16425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:34.113667 16407 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 07:06:34.113827 16407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-2/wal
uuid: "2272bdd8f6544fd4810d853858df7392"
format_stamp: "Formatted at 2026-08-07 07:06:34 on dist-test-slave-ncp9"
server_key: "86de166cb038dd29467632a4c7da0abb"
server_key_iv: "320fdd1b7364b3974d5328d911ef6c73"
server_key_version: "encryptionkey@0"
I20260807 07:06:34.114020 16407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:06:34.130224 16407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:06:34.134768 16407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:06:34.135072 16407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:06:34.136211 16407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:06:34.136649 16200 heartbeater.cc:503] Master 127.21.83.62:37755 was elected leader, sending a full tablet report...
I20260807 07:06:34.137992 16407 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:06:34.138100 16407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:34.138237 16407 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:06:34.138300 16407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:34.170373 16407 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.3:43943
I20260807 07:06:34.170410 16588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.3:43943 every 8 connection(s)
I20260807 07:06:34.171623 16407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
I20260807 07:06:34.180159 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 16407
I20260807 07:06:34.180359 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-2/wal/instance
I20260807 07:06:34.181001 21836 external_mini_cluster.cc:1428] Setting key acf43c469a12f7036c5c188eedf02091
Aug 07 07:06:34 dist-test-slave-ncp9 krb5kdc[15850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086394, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.3@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:06:34.191428 16592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:34.174030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:49801 (local address 127.21.83.62:37755)
0807 07:06:34.174549 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:34.174555 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:34.175542 (+ 987us) server_negotiation.cc:407] Connection header received
0807 07:06:34.177321 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:34.177326 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:34.177404 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:34.177490 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:34.178966 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:34.179766 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:34.181867 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:34.182409 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:34.182559 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:34.186117 (+ 3558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:34.186123 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:34.186126 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:06:34.186161 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:34.188712 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:34.189444 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:34.189449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:34.189451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:34.189591 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:34.189990 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:34.189993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:34.189995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:34.190255 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:34.190371 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:34.190976 (+ 605us) server_negotiation.cc:299] Negotiation successful
0807 07:06:34.191148 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":237,"threads_started":1}
I20260807 07:06:34.192332 16594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:34.174536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:37755 (local address 127.21.83.3:49801)
0807 07:06:34.175375 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:34.175410 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0807 07:06:34.177071 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:06:34.177688 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:06:34.177695 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:06:34.178168 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:06:34.178822 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:34.178840 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:34.179925 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:34.179931 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:34.181724 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:34.181740 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:34.182618 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:34.182623 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:34.182887 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:34.183638 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:06:34.183661 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:06:34.185903 (+ 2242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:06:34.188848 (+ 2945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:06:34.188854 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:06:34.188867 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:06:34.189331 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:06:34.189710 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:06:34.189714 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:06:34.189716 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:06:34.189905 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:06:34.190344 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:06:34.190350 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:06:34.190663 (+ 313us) client_negotiation.cc:769] Sending connection context
0807 07:06:34.190889 (+ 226us) client_negotiation.cc:240] Negotiation successful
0807 07:06:34.191273 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":336,"threads_started":1}
I20260807 07:06:34.193843 16589 heartbeater.cc:347] Connected to a master server at 127.21.83.62:37755
I20260807 07:06:34.194183 16589 heartbeater.cc:464] Registering TS with master...
I20260807 07:06:34.195122 16589 heartbeater.cc:511] Master 127.21.83.62:37755 requested a full tablet report, sending...
I20260807 07:06:34.196743 15917 ts_manager.cc:194] Registered new tserver with Master: 2272bdd8f6544fd4810d853858df7392 (127.21.83.3:43943)
I20260807 07:06:34.197624 15917 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.3@KRBTEST.COM'} at 127.21.83.3:49801
I20260807 07:06:34.208719 21836 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 07:06:34.224851 16592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:34.212791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60524 (local address 127.21.83.62:37755)
0807 07:06:34.212982 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:34.212987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:34.213350 (+ 363us) server_negotiation.cc:407] Connection header received
0807 07:06:34.213480 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:34.213484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:34.213560 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:34.213657 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:34.215071 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:34.215831 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:34.217100 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:34.217568 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:34.217674 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:34.220172 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:34.220177 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:34.220180 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:06:34.220211 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:34.222679 (+ 2468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:34.223328 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:34.223331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:34.223334 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:34.223445 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:34.223770 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:34.223773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:34.223775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:34.224048 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:34.224237 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:34.224487 (+ 250us) server_negotiation.cc:299] Negotiation successful
0807 07:06:34.224629 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 07:06:34.228372 15917 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60524:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 07:06:34.235280 15917 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 07:06:34.267191 16609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:34.246841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:41813 (local address 127.0.0.1:42922)
0807 07:06:34.258197 (+ 11356us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:34.258219 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 07:06:34.258482 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:06:34.259268 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:06:34.259275 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:06:34.259305 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:06:34.259839 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:34.259853 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:34.261931 (+ 2078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:34.261936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:34.264576 (+ 2640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:34.264591 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:34.266399 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:34.266405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:34.266567 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:34.266598 (+ 31us) client_negotiation.cc:769] Sending connection context
0807 07:06:34.266672 (+ 74us) client_negotiation.cc:240] Negotiation successful
0807 07:06:34.266783 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11196,"mutex_wait_us":96,"thread_start_us":11016,"threads_started":1}
I20260807 07:06:34.267760 16611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:34.246965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51214 (local address 127.21.83.3:43943)
0807 07:06:34.259450 (+ 12485us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:34.259458 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:34.259491 (+ 33us) server_negotiation.cc:407] Connection header received
0807 07:06:34.259594 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:34.259601 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:34.259813 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:34.260026 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:06:34.261143 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:34.262435 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:34.264733 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:34.265861 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:34.265995 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:34.266232 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:34.267286 (+ 1054us) server_negotiation.cc:299] Negotiation successful
0807 07:06:34.267443 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12307,"thread_start_us":12150,"threads_started":1}
I20260807 07:06:34.267967 16607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:34.247198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.2:34335 (local address 127.0.0.1:53444)
0807 07:06:34.259786 (+ 12588us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:34.259806 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 07:06:34.259925 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:06:34.260676 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:06:34.260681 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:06:34.260708 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:06:34.261129 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:34.261140 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:34.262730 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:34.262736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:34.265291 (+ 2555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:34.265306 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:34.267199 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:34.267205 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:34.267364 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:34.267395 (+ 31us) client_negotiation.cc:769] Sending connection context
0807 07:06:34.267470 (+ 75us) client_negotiation.cc:240] Negotiation successful
0807 07:06:34.267577 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12446,"mutex_wait_us":2,"thread_start_us":8540,"threads_started":1}
I20260807 07:06:34.268157 16613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:34.247757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42922 (local address 127.21.83.1:41813)
0807 07:06:34.257338 (+ 9581us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:34.257348 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:34.258610 (+ 1262us) server_negotiation.cc:407] Connection header received
0807 07:06:34.258726 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:34.258734 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:34.259007 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:34.260031 (+ 1024us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:06:34.260286 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:34.261785 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:34.264751 (+ 2966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:34.265916 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:34.266034 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:34.266254 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:34.267705 (+ 1451us) server_negotiation.cc:299] Negotiation successful
0807 07:06:34.267865 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9383,"thread_start_us":9195,"threads_started":1}
I20260807 07:06:34.268244 16612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:34.246630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.3:43943 (local address 127.0.0.1:51214)
0807 07:06:34.256262 (+ 9632us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:34.256320 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0807 07:06:34.256580 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:06:34.260309 (+ 3729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:06:34.260314 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:06:34.260339 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:06:34.260883 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:34.260922 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:34.262612 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:34.262617 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:34.264568 (+ 1951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:34.264588 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:34.266075 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:34.266082 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:34.266267 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:34.266328 (+ 61us) client_negotiation.cc:769] Sending connection context
0807 07:06:34.266455 (+ 127us) client_negotiation.cc:240] Negotiation successful
0807 07:06:34.266624 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9282,"mutex_wait_us":40,"thread_start_us":13781,"threads_started":1}
I20260807 07:06:34.268713 16615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:34.248139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53444 (local address 127.21.83.2:34335)
0807 07:06:34.257791 (+ 9652us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:34.257799 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:34.260034 (+ 2235us) server_negotiation.cc:407] Connection header received
0807 07:06:34.260143 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:34.260149 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:34.260341 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:34.260530 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:06:34.261276 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:34.262543 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:34.265453 (+ 2910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:34.266818 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:34.266934 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:34.267053 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:34.268262 (+ 1209us) server_negotiation.cc:299] Negotiation successful
0807 07:06:34.268417 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9499,"thread_start_us":9355,"threads_started":1}
I20260807 07:06:34.271509 16100 tablet_service.cc:1467] Processing CreateTablet for tablet 25f1634c3c8e420281238b5c820e2170 (DEFAULT_TABLE table=test-table [id=fce83d327e3f448f8f1cf39cce2d13e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:06:34.271787 16301 tablet_service.cc:1467] Processing CreateTablet for tablet 25f1634c3c8e420281238b5c820e2170 (DEFAULT_TABLE table=test-table [id=fce83d327e3f448f8f1cf39cce2d13e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:06:34.271790 16498 tablet_service.cc:1467] Processing CreateTablet for tablet 25f1634c3c8e420281238b5c820e2170 (DEFAULT_TABLE table=test-table [id=fce83d327e3f448f8f1cf39cce2d13e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:06:34.272878 16100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25f1634c3c8e420281238b5c820e2170. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:06:34.273077 16498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25f1634c3c8e420281238b5c820e2170. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:06:34.273178 16301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25f1634c3c8e420281238b5c820e2170. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:06:34.284631 16617 tablet_bootstrap.cc:492] T 25f1634c3c8e420281238b5c820e2170 P 53f31bf11eff49e2bf52819dac529388: Bootstrap starting.
I20260807 07:06:34.287022 16618 tablet_bootstrap.cc:492] T 25f1634c3c8e420281238b5c820e2170 P a43303fd25b849e485c8cc57c9f1fe58: Bootstrap starting.
I20260807 07:06:34.287807 16620 tablet_bootstrap.cc:492] T 25f1634c3c8e420281238b5c820e2170 P 2272bdd8f6544fd4810d853858df7392: Bootstrap starting.
I20260807 07:06:34.289249 16617 tablet_bootstrap.cc:654] T 25f1634c3c8e420281238b5c820e2170 P 53f31bf11eff49e2bf52819dac529388: Neither blocks nor log segments found. Creating new log.
I20260807 07:06:34.290791 16618 tablet_bootstrap.cc:654] T 25f1634c3c8e420281238b5c820e2170 P a43303fd25b849e485c8cc57c9f1fe58: Neither blocks nor log segments found. Creating new log.
I20260807 07:06:34.290555 16617 log.cc:824] T 25f1634c3c8e420281238b5c820e2170 P 53f31bf11eff49e2bf52819dac529388: Log is configured to *not* fsync() on all Append() calls
I20260807 07:06:34.291050 16620 tablet_bootstrap.cc:654] T 25f1634c3c8e420281238b5c820e2170 P 2272bdd8f6544fd4810d853858df7392: Neither blocks nor log segments found. Creating new log.
I20260807 07:06:34.291887 16618 log.cc:824] T 25f1634c3c8e420281238b5c820e2170 P a43303fd25b849e485c8cc57c9f1fe58: Log is configured to *not* fsync() on all Append() calls
I20260807 07:06:34.292181 16620 log.cc:824] T 25f1634c3c8e420281238b5c820e2170 P 2272bdd8f6544fd4810d853858df7392: Log is configured to *not* fsync() on all Append() calls
I20260807 07:06:34.294046 16617 tablet_bootstrap.cc:492] T 25f1634c3c8e420281238b5c820e2170 P 53f31bf11eff49e2bf52819dac529388: No bootstrap required, opened a new log
I20260807 07:06:34.294322 16617 ts_tablet_manager.cc:1397] T 25f1634c3c8e420281238b5c820e2170 P 53f31bf11eff49e2bf52819dac529388: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260807 07:06:34.295051 16620 tablet_bootstrap.cc:492] T 25f1634c3c8e420281238b5c820e2170 P 2272bdd8f6544fd4810d853858df7392: No bootstrap required, opened a new log
I20260807 07:06:34.295300 16620 ts_tablet_manager.cc:1397] T 25f1634c3c8e420281238b5c820e2170 P 2272bdd8f6544fd4810d853858df7392: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260807 07:06:34.298974 16618 tablet_bootstrap.cc:492] T 25f1634c3c8e420281238b5c820e2170 P a43303fd25b849e485c8cc57c9f1fe58: No bootstrap required, opened a new log
I20260807 07:06:34.299297 16618 ts_tablet_manager.cc:1397] T 25f1634c3c8e420281238b5c820e2170 P a43303fd25b849e485c8cc57c9f1fe58: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20260807 07:06:34.305298 16617 raft_consensus.cc:348] T 25f1634c3c8e420281238b5c820e2170 P 53f31bf11eff49e2bf52819dac529388 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2272bdd8f6544fd4810d853858df7392" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 43943 } } peers { permanent_uuid: "53f31bf11eff49e2bf52819dac529388" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 41813 } } peers { permanent_uuid: "a43303fd25b849e485c8cc57c9f1fe58" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 34335 } }
I20260807 07:06:34.305409 16620 raft_consensus.cc:348] T 25f1634c3c8e420281238b5c820e2170 P 2272bdd8f6544fd4810d853858df7392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2272bdd8f6544fd4810d853858df7392" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 43943 } } peers { permanent_uuid: "53f31bf11eff49e2bf52819dac529388" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 41813 } } peers { permanent_uuid: "a43303fd25b849e485c8cc57c9f1fe58" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 34335 } }
I20260807 07:06:34.305788 16617 raft_consensus.cc:374] T 25f1634c3c8e420281238b5c820e2170 P 53f31bf11eff49e2bf52819dac529388 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:06:34.305787 16620 raft_consensus.cc:374] T 25f1634c3c8e420281238b5c820e2170 P 2272bdd8f6544fd4810d853858df7392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:06:34.305893 16620 raft_consensus.cc:729] T 25f1634c3c8e420281238b5c820e2170 P 2272bdd8f6544fd4810d853858df7392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2272bdd8f6544fd4810d853858df7392, State: Initialized, Role: FOLLOWER
I20260807 07:06:34.305931 16617 raft_consensus.cc:729] T 25f1634c3c8e420281238b5c820e2170 P 53f31bf11eff49e2bf52819dac529388 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53f31bf11eff49e2bf52819dac529388, State: Initialized, Role: FOLLOWER
I20260807 07:06:34.306381 16618 raft_consensus.cc:348] T 25f1634c3c8e420281238b5c820e2170 P a43303fd25b849e485c8cc57c9f1fe58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2272bdd8f6544fd4810d853858df7392" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 43943 } } peers { permanent_uuid: "53f31bf11eff49e2bf52819dac529388" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 41813 } } peers { permanent_uuid: "a43303fd25b849e485c8cc57c9f1fe58" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 34335 } }
I20260807 07:06:34.306527 16620 consensus_queue.cc:260] T 25f1634c3c8e420281238b5c820e2170 P 2272bdd8f6544fd4810d853858df7392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2272bdd8f6544fd4810d853858df7392" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 43943 } } peers { permanent_uuid: "53f31bf11eff49e2bf52819dac529388" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 41813 } } peers { permanent_uuid: "a43303fd25b849e485c8cc57c9f1fe58" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 34335 } }
I20260807 07:06:34.306524 16617 consensus_queue.cc:260] T 25f1634c3c8e420281238b5c820e2170 P 53f31bf11eff49e2bf52819dac529388 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2272bdd8f6544fd4810d853858df7392" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 43943 } } peers { permanent_uuid: "53f31bf11eff49e2bf52819dac529388" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 41813 } } peers { permanent_uuid: "a43303fd25b849e485c8cc57c9f1fe58" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 34335 } }
I20260807 07:06:34.306733 16618 raft_consensus.cc:374] T 25f1634c3c8e420281238b5c820e2170 P a43303fd25b849e485c8cc57c9f1fe58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:06:34.306849 16618 raft_consensus.cc:729] T 25f1634c3c8e420281238b5c820e2170 P a43303fd25b849e485c8cc57c9f1fe58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a43303fd25b849e485c8cc57c9f1fe58, State: Initialized, Role: FOLLOWER
I20260807 07:06:34.307657 16589 heartbeater.cc:503] Master 127.21.83.62:37755 was elected leader, sending a full tablet report...
I20260807 07:06:34.307471 16618 consensus_queue.cc:260] T 25f1634c3c8e420281238b5c820e2170 P a43303fd25b849e485c8cc57c9f1fe58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2272bdd8f6544fd4810d853858df7392" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 43943 } } peers { permanent_uuid: "53f31bf11eff49e2bf52819dac529388" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 41813 } } peers { permanent_uuid: "a43303fd25b849e485c8cc57c9f1fe58" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 34335 } }
I20260807 07:06:34.309309 16620 ts_tablet_manager.cc:1428] T 25f1634c3c8e420281238b5c820e2170 P 2272bdd8f6544fd4810d853858df7392: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260807 07:06:34.310207 16398 heartbeater.cc:503] Master 127.21.83.62:37755 was elected leader, sending a full tablet report...
I20260807 07:06:34.310811 16618 ts_tablet_manager.cc:1428] T 25f1634c3c8e420281238b5c820e2170 P a43303fd25b849e485c8cc57c9f1fe58: Time spent starting tablet: real 0.011s user 0.008s sys 0.003s
I20260807 07:06:34.312811 16617 ts_tablet_manager.cc:1428] T 25f1634c3c8e420281238b5c820e2170 P 53f31bf11eff49e2bf52819dac529388: Time spent starting tablet: real 0.018s user 0.015s sys 0.000s
I20260807 07:06:34.335119 16625 raft_consensus.cc:482] T 25f1634c3c8e420281238b5c820e2170 P 2272bdd8f6544fd4810d853858df7392 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:06:34.335502 16625 raft_consensus.cc:504] T 25f1634c3c8e420281238b5c820e2170 P 2272bdd8f6544fd4810d853858df7392 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2272bdd8f6544fd4810d853858df7392" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 43943 } } peers { permanent_uuid: "53f31bf11eff49e2bf52819dac529388" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 41813 } } peers { permanent_uuid: "a43303fd25b849e485c8cc57c9f1fe58" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 34335 } }
I20260807 07:06:34.337342 16625 leader_election.cc:288] T 25f1634c3c8e420281238b5c820e2170 P 2272bdd8f6544fd4810d853858df7392 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53f31bf11eff49e2bf52819dac529388 (127.21.83.1:41813), a43303fd25b849e485c8cc57c9f1fe58 (127.21.83.2:34335)
I20260807 07:06:34.344274 16613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:34.337730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:54757 (local address 127.21.83.1:41813)
0807 07:06:34.338147 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:34.338153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:34.338177 (+ 24us) server_negotiation.cc:407] Connection header received
0807 07:06:34.338256 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:34.338261 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:34.338345 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:34.338438 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:06:34.338895 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:34.339867 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:34.342083 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:34.343212 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:34.343328 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:34.343411 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:34.343918 (+ 507us) server_negotiation.cc:299] Negotiation successful
0807 07:06:34.344039 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250}
I20260807 07:06:34.344331 16594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:34.337678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:41813 (local address 127.21.83.3:54757)
0807 07:06:34.337921 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:34.337944 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 07:06:34.338096 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:06:34.338497 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:06:34.338502 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:06:34.338531 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:06:34.338718 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:34.338728 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:34.340021 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:34.340027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:34.341913 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:34.341928 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:34.343383 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:34.343390 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:34.343687 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:34.343730 (+ 43us) client_negotiation.cc:769] Sending connection context
0807 07:06:34.343850 (+ 120us) client_negotiation.cc:240] Negotiation successful
0807 07:06:34.344004 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"mutex_wait_us":23}
I20260807 07:06:34.345211 16615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:34.338056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:56031 (local address 127.21.83.2:34335)
0807 07:06:34.338249 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:34.338254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:34.338800 (+ 546us) server_negotiation.cc:407] Connection header received
0807 07:06:34.338933 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:34.338938 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:34.338998 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:34.339063 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:06:34.339661 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:34.340629 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:34.342909 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:34.344092 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:34.344233 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:34.344330 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:34.344826 (+ 496us) server_negotiation.cc:299] Negotiation successful
0807 07:06:34.344999 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 07:06:34.345319 16628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:34.337822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.2:34335 (local address 127.21.83.3:56031)
0807 07:06:34.338695 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:34.338710 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 07:06:34.338811 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:06:34.339104 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:06:34.339109 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:06:34.339140 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:06:34.339510 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:34.339522 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:34.340825 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:34.340830 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:34.342714 (+ 1884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:34.342731 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:34.344345 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:34.344352 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:34.344545 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:34.344666 (+ 121us) client_negotiation.cc:769] Sending connection context
0807 07:06:34.344769 (+ 103us) client_negotiation.cc:240] Negotiation successful
0807 07:06:34.344878 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"mutex_wait_us":1,"thread_start_us":527,"threads_started":1}
I20260807 07:06:34.345531 16131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25f1634c3c8e420281238b5c820e2170" candidate_uuid: "2272bdd8f6544fd4810d853858df7392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53f31bf11eff49e2bf52819dac529388" is_pre_election: true
I20260807 07:06:34.346161 16131 raft_consensus.cc:2454] T 25f1634c3c8e420281238b5c820e2170 P 53f31bf11eff49e2bf52819dac529388 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2272bdd8f6544fd4810d853858df7392 in term 0.
I20260807 07:06:34.346457 16331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25f1634c3c8e420281238b5c820e2170" candidate_uuid: "2272bdd8f6544fd4810d853858df7392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a43303fd25b849e485c8cc57c9f1fe58" is_pre_election: true
I20260807 07:06:34.346953 16428 leader_election.cc:302] T 25f1634c3c8e420281238b5c820e2170 P 2272bdd8f6544fd4810d853858df7392 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2272bdd8f6544fd4810d853858df7392, 53f31bf11eff49e2bf52819dac529388; no voters:
I20260807 07:06:34.346954 16331 raft_consensus.cc:2454] T 25f1634c3c8e420281238b5c820e2170 P a43303fd25b849e485c8cc57c9f1fe58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2272bdd8f6544fd4810d853858df7392 in term 0.
I20260807 07:06:34.347352 16625 raft_consensus.cc:2781] T 25f1634c3c8e420281238b5c820e2170 P 2272bdd8f6544fd4810d853858df7392 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 07:06:34.347503 16625 raft_consensus.cc:482] T 25f1634c3c8e420281238b5c820e2170 P 2272bdd8f6544fd4810d853858df7392 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:06:34.347600 16625 raft_consensus.cc:3037] T 25f1634c3c8e420281238b5c820e2170 P 2272bdd8f6544fd4810d853858df7392 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:06:34.349972 16625 raft_consensus.cc:504] T 25f1634c3c8e420281238b5c820e2170 P 2272bdd8f6544fd4810d853858df7392 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2272bdd8f6544fd4810d853858df7392" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 43943 } } peers { permanent_uuid: "53f31bf11eff49e2bf52819dac529388" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 41813 } } peers { permanent_uuid: "a43303fd25b849e485c8cc57c9f1fe58" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 34335 } }
I20260807 07:06:34.350634 16625 leader_election.cc:288] T 25f1634c3c8e420281238b5c820e2170 P 2272bdd8f6544fd4810d853858df7392 [CANDIDATE]: Term 1 election: Requested vote from peers 53f31bf11eff49e2bf52819dac529388 (127.21.83.1:41813), a43303fd25b849e485c8cc57c9f1fe58 (127.21.83.2:34335)
I20260807 07:06:34.351138 16131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25f1634c3c8e420281238b5c820e2170" candidate_uuid: "2272bdd8f6544fd4810d853858df7392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53f31bf11eff49e2bf52819dac529388"
I20260807 07:06:34.351141 16331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25f1634c3c8e420281238b5c820e2170" candidate_uuid: "2272bdd8f6544fd4810d853858df7392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a43303fd25b849e485c8cc57c9f1fe58"
I20260807 07:06:34.351341 16331 raft_consensus.cc:3037] T 25f1634c3c8e420281238b5c820e2170 P a43303fd25b849e485c8cc57c9f1fe58 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:06:34.351349 16131 raft_consensus.cc:3037] T 25f1634c3c8e420281238b5c820e2170 P 53f31bf11eff49e2bf52819dac529388 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:06:34.353826 16331 raft_consensus.cc:2454] T 25f1634c3c8e420281238b5c820e2170 P a43303fd25b849e485c8cc57c9f1fe58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2272bdd8f6544fd4810d853858df7392 in term 1.
I20260807 07:06:34.353998 16131 raft_consensus.cc:2454] T 25f1634c3c8e420281238b5c820e2170 P 53f31bf11eff49e2bf52819dac529388 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2272bdd8f6544fd4810d853858df7392 in term 1.
I20260807 07:06:34.354486 16427 leader_election.cc:302] T 25f1634c3c8e420281238b5c820e2170 P 2272bdd8f6544fd4810d853858df7392 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2272bdd8f6544fd4810d853858df7392, a43303fd25b849e485c8cc57c9f1fe58; no voters:
I20260807 07:06:34.354943 16625 raft_consensus.cc:2781] T 25f1634c3c8e420281238b5c820e2170 P 2272bdd8f6544fd4810d853858df7392 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:06:34.355427 16625 raft_consensus.cc:686] T 25f1634c3c8e420281238b5c820e2170 P 2272bdd8f6544fd4810d853858df7392 [term 1 LEADER]: Becoming Leader. State: Replica: 2272bdd8f6544fd4810d853858df7392, State: Running, Role: LEADER
I20260807 07:06:34.355898 16625 consensus_queue.cc:237] T 25f1634c3c8e420281238b5c820e2170 P 2272bdd8f6544fd4810d853858df7392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2272bdd8f6544fd4810d853858df7392" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 43943 } } peers { permanent_uuid: "53f31bf11eff49e2bf52819dac529388" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 41813 } } peers { permanent_uuid: "a43303fd25b849e485c8cc57c9f1fe58" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 34335 } }
W20260807 07:06:34.360245 16201 tablet.cc:2369] T 25f1634c3c8e420281238b5c820e2170 P 53f31bf11eff49e2bf52819dac529388: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 07:06:34.361225 15916 catalog_manager.cc:5495] T 25f1634c3c8e420281238b5c820e2170 P 2272bdd8f6544fd4810d853858df7392 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2272bdd8f6544fd4810d853858df7392 (127.21.83.3). New cstate: current_term: 1 leader_uuid: "2272bdd8f6544fd4810d853858df7392" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2272bdd8f6544fd4810d853858df7392" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 43943 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53f31bf11eff49e2bf52819dac529388" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 41813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a43303fd25b849e485c8cc57c9f1fe58" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 34335 } health_report { overall_health: UNKNOWN } } }
W20260807 07:06:34.385746 16399 tablet.cc:2369] T 25f1634c3c8e420281238b5c820e2170 P a43303fd25b849e485c8cc57c9f1fe58: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 07:06:34.397286 16611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:34.392087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51228 (local address 127.21.83.3:43943)
0807 07:06:34.392258 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:34.392262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:34.392281 (+ 19us) server_negotiation.cc:407] Connection header received
0807 07:06:34.392414 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:34.392418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:34.392525 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:34.392650 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:06:34.393086 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:34.393843 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:34.395193 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:34.395582 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:34.395673 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:34.396198 (+ 525us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:06:34.396706 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:06:34.396839 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:34.397015 (+ 176us) server_negotiation.cc:299] Negotiation successful
0807 07:06:34.397060 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 07:06:34.405648 16131 raft_consensus.cc:1261] T 25f1634c3c8e420281238b5c820e2170 P 53f31bf11eff49e2bf52819dac529388 [term 1 FOLLOWER]: Refusing update from remote peer 2272bdd8f6544fd4810d853858df7392: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:06:34.405637 16331 raft_consensus.cc:1261] T 25f1634c3c8e420281238b5c820e2170 P a43303fd25b849e485c8cc57c9f1fe58 [term 1 FOLLOWER]: Refusing update from remote peer 2272bdd8f6544fd4810d853858df7392: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:06:34.406579 16631 consensus_queue.cc:1035] T 25f1634c3c8e420281238b5c820e2170 P 2272bdd8f6544fd4810d853858df7392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a43303fd25b849e485c8cc57c9f1fe58" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 34335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:06:34.406976 16625 consensus_queue.cc:1035] T 25f1634c3c8e420281238b5c820e2170 P 2272bdd8f6544fd4810d853858df7392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f31bf11eff49e2bf52819dac529388" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 41813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:06:34.419788 16635 mvcc.cc:205] Tried to move back new op lower bound from 7315809871475007488 to 7315809871286005760. Current Snapshot: MvccSnapshot[applied={T|T < 7315809871475007488}]
I20260807 07:06:34.423110 16638 mvcc.cc:205] Tried to move back new op lower bound from 7315809871475007488 to 7315809871286005760. Current Snapshot: MvccSnapshot[applied={T|T < 7315809871475007488}]
I20260807 07:06:34.434176 15913 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:06:34.434527 15913 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:06:34.438668 15913 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc25ec5f98a54be883a70158265dddd5: Sending DeleteTablet for 3 replicas of tablet 25f1634c3c8e420281238b5c820e2170
I20260807 07:06:34.439827 16498 tablet_service.cc:1514] Processing DeleteTablet for tablet 25f1634c3c8e420281238b5c820e2170 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:06:34 UTC) from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:51214
I20260807 07:06:34.439939 16100 tablet_service.cc:1514] Processing DeleteTablet for tablet 25f1634c3c8e420281238b5c820e2170 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:06:34 UTC) from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:42922
I20260807 07:06:34.439939 16301 tablet_service.cc:1514] Processing DeleteTablet for tablet 25f1634c3c8e420281238b5c820e2170 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:06:34 UTC) from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:53444
I20260807 07:06:34.444101 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 16000
I20260807 07:06:34.452291 16652 tablet_replica.cc:330] T 25f1634c3c8e420281238b5c820e2170 P 2272bdd8f6544fd4810d853858df7392: stopping tablet replica
I20260807 07:06:34.453163 16657 tablet_replica.cc:330] T 25f1634c3c8e420281238b5c820e2170 P 53f31bf11eff49e2bf52819dac529388: stopping tablet replica
I20260807 07:06:34.453135 16652 raft_consensus.cc:2229] T 25f1634c3c8e420281238b5c820e2170 P 2272bdd8f6544fd4810d853858df7392 [term 1 LEADER]: Raft consensus shutting down.
I20260807 07:06:34.453655 16657 raft_consensus.cc:2229] T 25f1634c3c8e420281238b5c820e2170 P 53f31bf11eff49e2bf52819dac529388 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:06:34.454032 16652 raft_consensus.cc:2258] T 25f1634c3c8e420281238b5c820e2170 P 2272bdd8f6544fd4810d853858df7392 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:06:34.454324 16657 raft_consensus.cc:2258] T 25f1634c3c8e420281238b5c820e2170 P 53f31bf11eff49e2bf52819dac529388 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:06:34.455240 16655 tablet_replica.cc:330] T 25f1634c3c8e420281238b5c820e2170 P a43303fd25b849e485c8cc57c9f1fe58: stopping tablet replica
I20260807 07:06:34.455721 16655 raft_consensus.cc:2229] T 25f1634c3c8e420281238b5c820e2170 P a43303fd25b849e485c8cc57c9f1fe58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:06:34.456292 16655 raft_consensus.cc:2258] T 25f1634c3c8e420281238b5c820e2170 P a43303fd25b849e485c8cc57c9f1fe58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:06:34.459187 16655 ts_tablet_manager.cc:1905] T 25f1634c3c8e420281238b5c820e2170 P a43303fd25b849e485c8cc57c9f1fe58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:06:34.459187 16657 ts_tablet_manager.cc:1905] T 25f1634c3c8e420281238b5c820e2170 P 53f31bf11eff49e2bf52819dac529388: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:06:34.466123 16657 ts_tablet_manager.cc:1918] T 25f1634c3c8e420281238b5c820e2170 P 53f31bf11eff49e2bf52819dac529388: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:06:34.466117 16655 ts_tablet_manager.cc:1918] T 25f1634c3c8e420281238b5c820e2170 P a43303fd25b849e485c8cc57c9f1fe58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:06:34.466281 16655 log.cc:1196] T 25f1634c3c8e420281238b5c820e2170 P a43303fd25b849e485c8cc57c9f1fe58: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-1/wal/wals/25f1634c3c8e420281238b5c820e2170
I20260807 07:06:34.466373 16657 log.cc:1196] T 25f1634c3c8e420281238b5c820e2170 P 53f31bf11eff49e2bf52819dac529388: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-0/wal/wals/25f1634c3c8e420281238b5c820e2170
I20260807 07:06:34.467001 16657 ts_tablet_manager.cc:1939] T 25f1634c3c8e420281238b5c820e2170 P 53f31bf11eff49e2bf52819dac529388: Deleting consensus metadata
I20260807 07:06:34.468952 15893 catalog_manager.cc:4841] TS 53f31bf11eff49e2bf52819dac529388 (127.21.83.1:41813): tablet 25f1634c3c8e420281238b5c820e2170 (table test-table [id=fce83d327e3f448f8f1cf39cce2d13e0]) successfully deleted
I20260807 07:06:34.469914 16655 ts_tablet_manager.cc:1939] T 25f1634c3c8e420281238b5c820e2170 P a43303fd25b849e485c8cc57c9f1fe58: Deleting consensus metadata
I20260807 07:06:34.470302 16652 ts_tablet_manager.cc:1905] T 25f1634c3c8e420281238b5c820e2170 P 2272bdd8f6544fd4810d853858df7392: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:06:34.471593 15892 catalog_manager.cc:4841] TS a43303fd25b849e485c8cc57c9f1fe58 (127.21.83.2:34335): tablet 25f1634c3c8e420281238b5c820e2170 (table test-table [id=fce83d327e3f448f8f1cf39cce2d13e0]) successfully deleted
Aug 07 07:06:34 dist-test-slave-ncp9 krb5kdc[15850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.1@KRBTEST.COM
I20260807 07:06:34.477690 16652 ts_tablet_manager.cc:1918] T 25f1634c3c8e420281238b5c820e2170 P 2272bdd8f6544fd4810d853858df7392: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:06:34.477964 16652 log.cc:1196] T 25f1634c3c8e420281238b5c820e2170 P 2272bdd8f6544fd4810d853858df7392: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786086078080656-21836-0/minicluster-data/ts-2/wal/wals/25f1634c3c8e420281238b5c820e2170
I20260807 07:06:34.478602 16652 ts_tablet_manager.cc:1939] T 25f1634c3c8e420281238b5c820e2170 P 2272bdd8f6544fd4810d853858df7392: Deleting consensus metadata
I20260807 07:06:34.480397 15895 catalog_manager.cc:4841] TS 2272bdd8f6544fd4810d853858df7392 (127.21.83.3:43943): tablet 25f1634c3c8e420281238b5c820e2170 (table test-table [id=fce83d327e3f448f8f1cf39cce2d13e0]) successfully deleted
I20260807 07:06:34.484443 16613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:34.445636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42932 (local address 127.21.83.1:41813)
0807 07:06:34.445842 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:34.445849 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:34.457205 (+ 11356us) server_negotiation.cc:407] Connection header received
0807 07:06:34.466775 (+ 9570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:34.466781 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:34.466868 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:34.466948 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:34.468792 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:34.469846 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:34.471782 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:34.472392 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:34.472481 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:34.477870 (+ 5389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:34.477884 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:34.477903 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:06:34.477942 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:34.481685 (+ 3743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:34.482577 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:34.482591 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:34.482593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:34.482721 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:34.483103 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:34.483105 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:34.483107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:34.483453 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:34.483697 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:34.484009 (+ 312us) server_negotiation.cc:299] Negotiation successful
0807 07:06:34.484175 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 07:06:34.485316 16188 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:06:34.575937 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 16000
I20260807 07:06:34.598274 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 16206
Aug 07 07:06:34 dist-test-slave-ncp9 krb5kdc[15850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.2@KRBTEST.COM
I20260807 07:06:34.613756 16615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:34.599084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53460 (local address 127.21.83.2:34335)
0807 07:06:34.599241 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:34.599245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:34.599438 (+ 193us) server_negotiation.cc:407] Connection header received
0807 07:06:34.599618 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:34.599624 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:34.599697 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:34.599782 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:34.601090 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:34.601878 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:34.603259 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:34.603665 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:34.603775 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:34.608125 (+ 4350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:34.608139 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:34.608153 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:06:34.608187 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:34.611019 (+ 2832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:34.611856 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:34.611863 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:34.611865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:34.611989 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:34.612349 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:34.612352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:34.612354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:34.612741 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:34.613017 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:34.613301 (+ 284us) server_negotiation.cc:299] Negotiation successful
0807 07:06:34.613462 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260807 07:06:34.614369 16389 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:06:34.705550 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 16206
I20260807 07:06:34.735378 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 16407
Aug 07 07:06:34 dist-test-slave-ncp9 krb5kdc[15850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.3@KRBTEST.COM
I20260807 07:06:34.754194 16611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:34.736188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51236 (local address 127.21.83.3:43943)
0807 07:06:34.736346 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:34.736350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:34.736371 (+ 21us) server_negotiation.cc:407] Connection header received
0807 07:06:34.736608 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:34.736614 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:34.736688 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:34.736786 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:34.738621 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:34.739475 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:34.741403 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:34.741876 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:34.741980 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:34.747613 (+ 5633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:34.747638 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:34.747654 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:06:34.747710 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:34.750850 (+ 3140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:34.751687 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:34.751696 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:34.751698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:34.751889 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:34.752329 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:34.752333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:34.752335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:34.752748 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:34.753090 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:34.753439 (+ 349us) server_negotiation.cc:299] Negotiation successful
0807 07:06:34.753823 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260807 07:06:34.755012 16584 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:06:34.880764 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 16407
I20260807 07:06:34.906735 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 15869
I20260807 07:06:34.907899 15956 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:06:35.020442 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 15869
2026-08-07T07:06:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4612 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-07T07:06:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:06:35Z Disabled control of system clock
I20260807 07:06:35.126408 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:36201
--webserver_interface=127.21.83.62
--webserver_port=0
--builtin_ntp_servers=127.21.83.20:34225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:36201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 07:06:35.472950 16669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:06:35.473289 16669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:06:35.473420 16669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:06:35.481068 16669 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:06:35.481211 16669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:06:35.481276 16669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:06:35.481353 16669 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:06:35.481413 16669 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:06:35.490836 16669 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:34225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:36201
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:36201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:06:35.492092 16669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:06:35.493896 16669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:06:35.505242 16674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:35.505354 16675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:35.505439 16678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:06:35.506356 16669 server_base.cc:1034] running on GCE node
I20260807 07:06:35.507117 16669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:06:35.508435 16669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:06:35.509624 16669 hybrid_clock.cc:648] HybridClock initialized: now 1786086395509584 us; error 33 us; skew 500 ppm
I20260807 07:06:35.509931 16669 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 07:06:35.512151 16669 webserver.cc:460] Webserver started at http://127.21.83.62:43115/ using document root <none> and password file <none>
I20260807 07:06:35.512810 16669 fs_manager.cc:359] Metadata directory not provided
I20260807 07:06:35.512920 16669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:06:35.513226 16669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:06:35.516113 16669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "1173183a0b3e48c48e1b06a3143f5e5a"
format_stamp: "Formatted at 2026-08-07 07:06:35 on dist-test-slave-ncp9"
server_key: "2ef7d58ec0f1772a873e527f10f63dff"
server_key_iv: "d5644148b413969ccd7c87d158f8fb33"
server_key_version: "encryptionkey@0"
I20260807 07:06:35.516952 16669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "1173183a0b3e48c48e1b06a3143f5e5a"
format_stamp: "Formatted at 2026-08-07 07:06:35 on dist-test-slave-ncp9"
server_key: "2ef7d58ec0f1772a873e527f10f63dff"
server_key_iv: "d5644148b413969ccd7c87d158f8fb33"
server_key_version: "encryptionkey@0"
I20260807 07:06:35.522416 16669 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 07:06:35.526228 16692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:35.527689 16669 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260807 07:06:35.527871 16669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "1173183a0b3e48c48e1b06a3143f5e5a"
format_stamp: "Formatted at 2026-08-07 07:06:35 on dist-test-slave-ncp9"
server_key: "2ef7d58ec0f1772a873e527f10f63dff"
server_key_iv: "d5644148b413969ccd7c87d158f8fb33"
server_key_version: "encryptionkey@0"
I20260807 07:06:35.528079 16669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:06:35.541709 16669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:06:35.542555 16669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:06:35.542806 16669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:06:35.562757 16669 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.62:36201
I20260807 07:06:35.562790 16767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.62:36201 every 8 connection(s)
I20260807 07:06:35.564292 16669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:06:35.568356 16770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:06:35.570516 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 16669
I20260807 07:06:35.570689 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:06:35.571183 21836 external_mini_cluster.cc:1428] Setting key 04ddffa4eadb5d00ad1478553adc17d5
I20260807 07:06:35.584103 16770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1173183a0b3e48c48e1b06a3143f5e5a: Bootstrap starting.
I20260807 07:06:35.587141 16774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:35.573441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54210 (local address 127.21.83.62:36201)
0807 07:06:35.574953 (+ 1512us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:35.574968 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:35.575026 (+ 58us) server_negotiation.cc:407] Connection header received
0807 07:06:35.577534 (+ 2508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:35.577569 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:35.578226 (+ 657us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:35.578670 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:35.579365 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:35.580793 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:35.582315 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:35.583090 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:35.583271 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:35.584528 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:35.584536 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:35.584551 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:06:35.585319 (+ 768us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:35.585409 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:06:35.585434 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:35.585549 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:35.585801 (+ 252us) server_negotiation.cc:299] Negotiation successful
0807 07:06:35.586147 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":952,"thread_start_us":566,"threads_started":1}
I20260807 07:06:35.588266 16770 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1173183a0b3e48c48e1b06a3143f5e5a: Neither blocks nor log segments found. Creating new log.
I20260807 07:06:35.589448 16770 log.cc:824] T 00000000000000000000000000000000 P 1173183a0b3e48c48e1b06a3143f5e5a: Log is configured to *not* fsync() on all Append() calls
I20260807 07:06:35.592340 16770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1173183a0b3e48c48e1b06a3143f5e5a: No bootstrap required, opened a new log
I20260807 07:06:35.600193 16770 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1173183a0b3e48c48e1b06a3143f5e5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1173183a0b3e48c48e1b06a3143f5e5a" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 36201 } }
I20260807 07:06:35.600549 16770 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1173183a0b3e48c48e1b06a3143f5e5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:06:35.600621 16770 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1173183a0b3e48c48e1b06a3143f5e5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1173183a0b3e48c48e1b06a3143f5e5a, State: Initialized, Role: FOLLOWER
I20260807 07:06:35.601176 16770 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1173183a0b3e48c48e1b06a3143f5e5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1173183a0b3e48c48e1b06a3143f5e5a" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 36201 } }
I20260807 07:06:35.601406 16770 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1173183a0b3e48c48e1b06a3143f5e5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:06:35.601508 16770 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1173183a0b3e48c48e1b06a3143f5e5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:06:35.601631 16770 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1173183a0b3e48c48e1b06a3143f5e5a [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:06:35.603762 16770 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1173183a0b3e48c48e1b06a3143f5e5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1173183a0b3e48c48e1b06a3143f5e5a" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 36201 } }
I20260807 07:06:35.604216 16770 leader_election.cc:302] T 00000000000000000000000000000000 P 1173183a0b3e48c48e1b06a3143f5e5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1173183a0b3e48c48e1b06a3143f5e5a; no voters:
I20260807 07:06:35.604779 16770 leader_election.cc:288] T 00000000000000000000000000000000 P 1173183a0b3e48c48e1b06a3143f5e5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 07:06:35.604938 16777 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1173183a0b3e48c48e1b06a3143f5e5a [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:06:35.606045 16777 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1173183a0b3e48c48e1b06a3143f5e5a [term 1 LEADER]: Becoming Leader. State: Replica: 1173183a0b3e48c48e1b06a3143f5e5a, State: Running, Role: LEADER
I20260807 07:06:35.606544 16777 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1173183a0b3e48c48e1b06a3143f5e5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1173183a0b3e48c48e1b06a3143f5e5a" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 36201 } }
I20260807 07:06:35.606892 16770 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1173183a0b3e48c48e1b06a3143f5e5a [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:06:35.609133 16778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1173183a0b3e48c48e1b06a3143f5e5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1173183a0b3e48c48e1b06a3143f5e5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1173183a0b3e48c48e1b06a3143f5e5a" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 36201 } } }
I20260807 07:06:35.609081 16779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1173183a0b3e48c48e1b06a3143f5e5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1173183a0b3e48c48e1b06a3143f5e5a. Latest consensus state: current_term: 1 leader_uuid: "1173183a0b3e48c48e1b06a3143f5e5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1173183a0b3e48c48e1b06a3143f5e5a" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 36201 } } }
I20260807 07:06:35.609542 16779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1173183a0b3e48c48e1b06a3143f5e5a [sys.catalog]: This master's current role is: LEADER
I20260807 07:06:35.609547 16778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1173183a0b3e48c48e1b06a3143f5e5a [sys.catalog]: This master's current role is: LEADER
I20260807 07:06:35.611217 16784 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:06:35.616540 16784 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:06:35.624536 16784 catalog_manager.cc:1347] Generated new cluster ID: cc29d68abc3d4215bf0b92e7778f2a8e
I20260807 07:06:35.624678 16784 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:06:35.650569 16784 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:06:35.651952 16784 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:06:35.662326 16784 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1173183a0b3e48c48e1b06a3143f5e5a: Generated new TSK 0
I20260807 07:06:35.663300 16784 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 07:06:35.674182 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.1:0
--local_ip_for_outbound_sockets=127.21.83.1
--webserver_interface=127.21.83.1
--webserver_port=0
--tserver_master_addrs=127.21.83.62:36201
--builtin_ntp_servers=127.21.83.20:34225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 07:06:36.041013 16801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:06:36.041374 16801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:06:36.041523 16801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:06:36.049368 16801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:06:36.049504 16801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:06:36.049804 16801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.1
I20260807 07:06:36.059634 16801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:34225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.83.1
--webserver_port=0
--tserver_master_addrs=127.21.83.62:36201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.1
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:06:36.061064 16801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:06:36.063004 16801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:06:36.076586 16809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:36.076555 16814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:36.076505 16811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:06:36.077349 16801 server_base.cc:1034] running on GCE node
I20260807 07:06:36.078104 16801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:06:36.079115 16801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:06:36.080327 16801 hybrid_clock.cc:648] HybridClock initialized: now 1786086396080273 us; error 47 us; skew 500 ppm
I20260807 07:06:36.080657 16801 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 07:06:36.083166 16801 webserver.cc:460] Webserver started at http://127.21.83.1:44947/ using document root <none> and password file <none>
I20260807 07:06:36.083880 16801 fs_manager.cc:359] Metadata directory not provided
I20260807 07:06:36.083990 16801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:06:36.084426 16801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:06:36.087682 16801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-0/data/instance:
uuid: "aa299af27c974ca4b661a728eee5b963"
format_stamp: "Formatted at 2026-08-07 07:06:36 on dist-test-slave-ncp9"
server_key: "48133b70652edf67de793479c063ddcb"
server_key_iv: "ccdaa4bcfcb53bf782cc026b90756802"
server_key_version: "encryptionkey@0"
I20260807 07:06:36.088479 16801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-0/wal/instance:
uuid: "aa299af27c974ca4b661a728eee5b963"
format_stamp: "Formatted at 2026-08-07 07:06:36 on dist-test-slave-ncp9"
server_key: "48133b70652edf67de793479c063ddcb"
server_key_iv: "ccdaa4bcfcb53bf782cc026b90756802"
server_key_version: "encryptionkey@0"
I20260807 07:06:36.093513 16801 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260807 07:06:36.096866 16823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:36.098429 16801 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 07:06:36.098564 16801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-0/wal
uuid: "aa299af27c974ca4b661a728eee5b963"
format_stamp: "Formatted at 2026-08-07 07:06:36 on dist-test-slave-ncp9"
server_key: "48133b70652edf67de793479c063ddcb"
server_key_iv: "ccdaa4bcfcb53bf782cc026b90756802"
server_key_version: "encryptionkey@0"
I20260807 07:06:36.098738 16801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:06:36.114147 16801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:06:36.114923 16801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:06:36.115172 16801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:06:36.116115 16801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:06:36.117681 16801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:06:36.117769 16801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:36.117854 16801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:06:36.117909 16801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:36.149513 16801 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.1:38337
I20260807 07:06:36.149700 16993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.1:38337 every 8 connection(s)
I20260807 07:06:36.150967 16801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
I20260807 07:06:36.153695 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 16801
I20260807 07:06:36.153909 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-0/wal/instance
I20260807 07:06:36.154429 21836 external_mini_cluster.cc:1428] Setting key 6239115a4f04f54df4531e53ea49f7e1
I20260807 07:06:36.157864 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.2:0
--local_ip_for_outbound_sockets=127.21.83.2
--webserver_interface=127.21.83.2
--webserver_port=0
--tserver_master_addrs=127.21.83.62:36201
--builtin_ntp_servers=127.21.83.20:34225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 07:06:36.166003 16996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:36.153720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:54217 (local address 127.21.83.62:36201)
0807 07:06:36.154265 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:36.154272 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:36.155738 (+ 1466us) server_negotiation.cc:407] Connection header received
0807 07:06:36.158203 (+ 2465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:36.158207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:36.158283 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:36.158376 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:36.159911 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:36.160676 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:36.162898 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:36.163320 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:36.163428 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:36.165099 (+ 1671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:36.165105 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:36.165108 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:06:36.165132 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:36.165176 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:06:36.165192 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:36.165274 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:36.165608 (+ 334us) server_negotiation.cc:299] Negotiation successful
0807 07:06:36.165710 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":252,"threads_started":1}
I20260807 07:06:36.167323 16998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:36.154342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:36201 (local address 127.21.83.1:54217)
0807 07:06:36.155481 (+ 1139us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:36.155530 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 07:06:36.157822 (+ 2292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:06:36.158676 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:06:36.158688 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:06:36.158817 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:06:36.159681 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:36.159710 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:36.160846 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:36.160851 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:36.162738 (+ 1887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:36.162755 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:36.163562 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:36.163570 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:36.163851 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:36.164668 (+ 817us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 07:06:36.164671 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:06:36.164739 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 07:06:36.164851 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:06:36.165288 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:06:36.165300 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:06:36.165308 (+ 8us) client_negotiation.cc:769] Sending connection context
0807 07:06:36.165505 (+ 197us) client_negotiation.cc:240] Negotiation successful
0807 07:06:36.165952 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":348,"threads_started":1}
I20260807 07:06:36.169175 16994 heartbeater.cc:347] Connected to a master server at 127.21.83.62:36201
I20260807 07:06:36.169540 16994 heartbeater.cc:464] Registering TS with master...
I20260807 07:06:36.170501 16994 heartbeater.cc:511] Master 127.21.83.62:36201 requested a full tablet report, sending...
I20260807 07:06:36.173008 16719 ts_manager.cc:194] Registered new tserver with Master: aa299af27c974ca4b661a728eee5b963 (127.21.83.1:38337)
I20260807 07:06:36.175062 16719 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.83.1:54217
W20260807 07:06:36.505553 16999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:06:36.505860 16999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:06:36.505990 16999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:06:36.513459 16999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:06:36.513576 16999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:06:36.513732 16999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.2
I20260807 07:06:36.523451 16999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:34225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.83.2
--webserver_port=0
--tserver_master_addrs=127.21.83.62:36201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.2
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:06:36.524813 16999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:06:36.526655 16999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:06:36.540596 17005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:36.540967 17007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:06:36.542006 16999 server_base.cc:1034] running on GCE node
W20260807 07:06:36.540568 17009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:06:36.543977 16999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:06:36.545249 16999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:06:36.546491 16999 hybrid_clock.cc:648] HybridClock initialized: now 1786086396546423 us; error 54 us; skew 500 ppm
I20260807 07:06:36.546869 16999 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 07:06:36.549296 16999 webserver.cc:460] Webserver started at http://127.21.83.2:36963/ using document root <none> and password file <none>
I20260807 07:06:36.550035 16999 fs_manager.cc:359] Metadata directory not provided
I20260807 07:06:36.550139 16999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:06:36.550470 16999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:06:36.553550 16999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-1/data/instance:
uuid: "8f80383906244d3c936b7e3da3f122b7"
format_stamp: "Formatted at 2026-08-07 07:06:36 on dist-test-slave-ncp9"
server_key: "70a89168ac2120507451df29066787b4"
server_key_iv: "85fd7caa306678cc11e85aff41b2fb7f"
server_key_version: "encryptionkey@0"
I20260807 07:06:36.554382 16999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-1/wal/instance:
uuid: "8f80383906244d3c936b7e3da3f122b7"
format_stamp: "Formatted at 2026-08-07 07:06:36 on dist-test-slave-ncp9"
server_key: "70a89168ac2120507451df29066787b4"
server_key_iv: "85fd7caa306678cc11e85aff41b2fb7f"
server_key_version: "encryptionkey@0"
I20260807 07:06:36.559880 16999 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 07:06:36.564236 17018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:36.565781 16999 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260807 07:06:36.565951 16999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-1/wal
uuid: "8f80383906244d3c936b7e3da3f122b7"
format_stamp: "Formatted at 2026-08-07 07:06:36 on dist-test-slave-ncp9"
server_key: "70a89168ac2120507451df29066787b4"
server_key_iv: "85fd7caa306678cc11e85aff41b2fb7f"
server_key_version: "encryptionkey@0"
I20260807 07:06:36.566169 16999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:06:36.577649 16999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:06:36.578390 16999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:06:36.578641 16999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:06:36.579586 16999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:06:36.581009 16999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:06:36.581110 16999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:36.581218 16999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:06:36.581284 16999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:36.611960 16999 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.2:39811
I20260807 07:06:36.612042 17179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.2:39811 every 8 connection(s)
I20260807 07:06:36.613304 16999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
I20260807 07:06:36.615052 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 16999
I20260807 07:06:36.615224 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-1/wal/instance
I20260807 07:06:36.615708 21836 external_mini_cluster.cc:1428] Setting key 5a82bb42860b0a7a5e7bf5032c4dad9e
I20260807 07:06:36.618916 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.3:0
--local_ip_for_outbound_sockets=127.21.83.3
--webserver_interface=127.21.83.3
--webserver_port=0
--tserver_master_addrs=127.21.83.62:36201
--builtin_ntp_servers=127.21.83.20:34225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 07:06:36.626704 16996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:36.615841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:42893 (local address 127.21.83.62:36201)
0807 07:06:36.616054 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:36.616060 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:36.617470 (+ 1410us) server_negotiation.cc:407] Connection header received
0807 07:06:36.619781 (+ 2311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:36.619784 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:36.619848 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:36.619931 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:36.621277 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:36.622009 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:36.624011 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:36.624408 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:36.624520 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:36.626007 (+ 1487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:36.626013 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:36.626016 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:06:36.626036 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:36.626069 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:06:36.626081 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:36.626148 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:36.626372 (+ 224us) server_negotiation.cc:299] Negotiation successful
0807 07:06:36.626455 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 07:06:36.627962 17183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:36.616317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:36201 (local address 127.21.83.2:42893)
0807 07:06:36.617313 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:36.617354 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 07:06:36.619505 (+ 2151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:06:36.620177 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:06:36.620187 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:06:36.620302 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:06:36.621103 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:36.621129 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:36.622161 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:36.622166 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:36.623851 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:36.623866 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:36.624630 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:36.624636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:36.624923 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:36.625667 (+ 744us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 07:06:36.625670 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:06:36.625728 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 07:06:36.625821 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:06:36.626162 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:06:36.626172 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:06:36.626180 (+ 8us) client_negotiation.cc:769] Sending connection context
0807 07:06:36.626350 (+ 170us) client_negotiation.cc:240] Negotiation successful
0807 07:06:36.626722 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":339,"threads_started":1}
I20260807 07:06:36.629369 17180 heartbeater.cc:347] Connected to a master server at 127.21.83.62:36201
I20260807 07:06:36.629654 17180 heartbeater.cc:464] Registering TS with master...
I20260807 07:06:36.630337 17180 heartbeater.cc:511] Master 127.21.83.62:36201 requested a full tablet report, sending...
I20260807 07:06:36.631644 16719 ts_manager.cc:194] Registered new tserver with Master: 8f80383906244d3c936b7e3da3f122b7 (127.21.83.2:39811)
I20260807 07:06:36.632479 16719 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.83.2:42893
W20260807 07:06:36.971272 17184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:06:36.971599 17184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:06:36.971696 17184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:06:36.979657 17184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:06:36.979789 17184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:06:36.979902 17184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.3
I20260807 07:06:36.989460 17184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:34225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.83.3
--webserver_port=0
--tserver_master_addrs=127.21.83.62:36201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.3
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:06:36.990757 17184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:06:36.992509 17184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:06:37.006065 17196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:37.006085 17192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:37.006104 17193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:06:37.006971 17184 server_base.cc:1034] running on GCE node
I20260807 07:06:37.007686 17184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:06:37.008724 17184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:06:37.009972 17184 hybrid_clock.cc:648] HybridClock initialized: now 1786086397009889 us; error 55 us; skew 500 ppm
I20260807 07:06:37.010294 17184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 07:06:37.012760 17184 webserver.cc:460] Webserver started at http://127.21.83.3:46241/ using document root <none> and password file <none>
I20260807 07:06:37.013516 17184 fs_manager.cc:359] Metadata directory not provided
I20260807 07:06:37.013612 17184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:06:37.013888 17184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:06:37.016817 17184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-2/data/instance:
uuid: "36b9b64beecb47c5958dd5bf4058ff88"
format_stamp: "Formatted at 2026-08-07 07:06:37 on dist-test-slave-ncp9"
server_key: "f63f061858eb21b210cb5324481755f3"
server_key_iv: "fb62755a0dfedf4ea82ee7028a2cc708"
server_key_version: "encryptionkey@0"
I20260807 07:06:37.017642 17184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-2/wal/instance:
uuid: "36b9b64beecb47c5958dd5bf4058ff88"
format_stamp: "Formatted at 2026-08-07 07:06:37 on dist-test-slave-ncp9"
server_key: "f63f061858eb21b210cb5324481755f3"
server_key_iv: "fb62755a0dfedf4ea82ee7028a2cc708"
server_key_version: "encryptionkey@0"
I20260807 07:06:37.023085 17184 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 07:06:37.027196 17204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:37.028653 17184 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 07:06:37.028826 17184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-2/wal
uuid: "36b9b64beecb47c5958dd5bf4058ff88"
format_stamp: "Formatted at 2026-08-07 07:06:37 on dist-test-slave-ncp9"
server_key: "f63f061858eb21b210cb5324481755f3"
server_key_iv: "fb62755a0dfedf4ea82ee7028a2cc708"
server_key_version: "encryptionkey@0"
I20260807 07:06:37.029076 17184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:06:37.053194 17184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:06:37.054184 17184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:06:37.054473 17184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:06:37.055584 17184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:06:37.057315 17184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:06:37.057423 17184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:37.057530 17184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:06:37.057605 17184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:37.090338 17184 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.3:41585
I20260807 07:06:37.090406 17375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.3:41585 every 8 connection(s)
I20260807 07:06:37.091686 17184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
I20260807 07:06:37.095167 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 17184
I20260807 07:06:37.095548 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-2/wal/instance
I20260807 07:06:37.096158 21836 external_mini_cluster.cc:1428] Setting key dc152c3272c10b983ae1790e623d7fd9
I20260807 07:06:37.104494 16996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:37.094310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:58141 (local address 127.21.83.62:36201)
0807 07:06:37.094483 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:37.094488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:37.096033 (+ 1545us) server_negotiation.cc:407] Connection header received
0807 07:06:37.097894 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:37.097899 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:37.097988 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:37.098092 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:37.099352 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:37.100119 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:37.101741 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:37.102178 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:37.102322 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:37.103745 (+ 1423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:37.103751 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:37.103755 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:06:37.103776 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:37.103812 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:06:37.103824 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:37.103904 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:37.104131 (+ 227us) server_negotiation.cc:299] Negotiation successful
0807 07:06:37.104231 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 07:06:37.105531 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:37.094757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:36201 (local address 127.21.83.3:58141)
0807 07:06:37.095847 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:37.095892 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 07:06:37.097636 (+ 1744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:06:37.098360 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:06:37.098369 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:06:37.098469 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:06:37.099175 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:37.099196 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:37.100267 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:37.100271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:37.101595 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:37.101609 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:37.102447 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:37.102452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:37.102685 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:37.103392 (+ 707us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 07:06:37.103395 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:06:37.103450 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 07:06:37.103551 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:06:37.103908 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:06:37.103914 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:06:37.103920 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 07:06:37.104063 (+ 143us) client_negotiation.cc:240] Negotiation successful
0807 07:06:37.104400 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":335,"threads_started":1}
I20260807 07:06:37.106961 17379 heartbeater.cc:347] Connected to a master server at 127.21.83.62:36201
I20260807 07:06:37.107266 17379 heartbeater.cc:464] Registering TS with master...
I20260807 07:06:37.108044 17379 heartbeater.cc:511] Master 127.21.83.62:36201 requested a full tablet report, sending...
I20260807 07:06:37.109592 16719 ts_manager.cc:194] Registered new tserver with Master: 36b9b64beecb47c5958dd5bf4058ff88 (127.21.83.3:41585)
I20260807 07:06:37.110549 16719 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.83.3:58141
I20260807 07:06:37.117112 21836 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 07:06:37.126776 16996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:37.121238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54214 (local address 127.21.83.62:36201)
0807 07:06:37.121500 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:37.121507 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:37.121800 (+ 293us) server_negotiation.cc:407] Connection header received
0807 07:06:37.121950 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:37.121954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:37.122032 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:37.122112 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:37.122543 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:37.123266 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:37.124609 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:37.125088 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:37.125185 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:37.126124 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:37.126128 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:37.126131 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:06:37.126150 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:37.126182 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:06:37.126193 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:37.126291 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:37.126410 (+ 119us) server_negotiation.cc:299] Negotiation successful
0807 07:06:37.126460 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 07:06:37.130681 16717 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54214:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 07:06:37.137864 16717 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 07:06:37.178401 16994 heartbeater.cc:503] Master 127.21.83.62:36201 was elected leader, sending a full tablet report...
I20260807 07:06:37.179390 17402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:37.152079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56564 (local address 127.21.83.2:39811)
0807 07:06:37.167508 (+ 15429us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:37.167519 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:37.169720 (+ 2201us) server_negotiation.cc:407] Connection header received
0807 07:06:37.169968 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:37.169984 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:37.170368 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:37.170659 (+ 291us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:06:37.171713 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:37.173176 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:37.176568 (+ 3392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:37.178002 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:37.178140 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:37.178353 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:37.178866 (+ 513us) server_negotiation.cc:299] Negotiation successful
0807 07:06:37.179048 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15211,"thread_start_us":16936,"threads_started":1}
I20260807 07:06:37.179459 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:37.150668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.2:39811 (local address 127.0.0.1:56564)
0807 07:06:37.169588 (+ 18920us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:37.169606 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 07:06:37.169797 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:06:37.170944 (+ 1147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:06:37.170949 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:06:37.170991 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:06:37.171510 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:37.171525 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:37.173307 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:37.173313 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:37.176361 (+ 3048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:37.176375 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:37.178462 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:37.178467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:37.178623 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:37.178671 (+ 48us) client_negotiation.cc:769] Sending connection context
0807 07:06:37.178795 (+ 124us) client_negotiation.cc:240] Negotiation successful
0807 07:06:37.179019 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18780,"mutex_wait_us":163,"thread_start_us":13654,"threads_started":1}
I20260807 07:06:37.179541 17399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:37.150143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:38337 (local address 127.0.0.1:39752)
0807 07:06:37.163765 (+ 13622us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:37.163840 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0807 07:06:37.164074 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:06:37.169999 (+ 5925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:06:37.170006 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:06:37.170038 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:06:37.170494 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:37.170511 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:37.173133 (+ 2622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:37.173138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:37.175882 (+ 2744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:37.175902 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:37.178000 (+ 2098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:37.178007 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:37.178844 (+ 837us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:37.178879 (+ 35us) client_negotiation.cc:769] Sending connection context
0807 07:06:37.178967 (+ 88us) client_negotiation.cc:240] Negotiation successful
0807 07:06:37.179111 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13256,"mutex_wait_us":121,"thread_start_us":19183,"threads_started":1}
I20260807 07:06:37.180540 17404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:37.169774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.3:41585 (local address 127.0.0.1:53022)
0807 07:06:37.171499 (+ 1725us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:37.171514 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 07:06:37.171642 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:06:37.172164 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:06:37.172169 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:06:37.172193 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:06:37.172524 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:37.172534 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:37.174270 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:37.174275 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:37.176223 (+ 1948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:37.176239 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:37.179776 (+ 3537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:37.179783 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:37.179959 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:37.179990 (+ 31us) client_negotiation.cc:769] Sending connection context
0807 07:06:37.180070 (+ 80us) client_negotiation.cc:240] Negotiation successful
0807 07:06:37.180198 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1613,"mutex_wait_us":24,"thread_start_us":1412,"threads_started":1}
I20260807 07:06:37.181360 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:37.151540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39752 (local address 127.21.83.1:38337)
0807 07:06:37.167995 (+ 16455us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:37.168004 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:37.168040 (+ 36us) server_negotiation.cc:407] Connection header received
0807 07:06:37.168155 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:37.168163 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:37.168489 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:37.168728 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:06:37.171252 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:37.172966 (+ 1714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:37.176153 (+ 3187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:37.177775 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:37.177932 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:37.178259 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:37.180787 (+ 2528us) server_negotiation.cc:299] Negotiation successful
0807 07:06:37.181008 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16238,"thread_start_us":17450,"threads_started":1}
I20260807 07:06:37.181990 17403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:37.165746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53022 (local address 127.21.83.3:41585)
0807 07:06:37.171131 (+ 5385us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:37.171139 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:37.171631 (+ 492us) server_negotiation.cc:407] Connection header received
0807 07:06:37.171803 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:37.171809 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:37.171973 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:37.172139 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:06:37.172704 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:37.174135 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:37.176447 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:37.177795 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:37.177927 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:37.178235 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:37.181432 (+ 3197us) server_negotiation.cc:299] Negotiation successful
0807 07:06:37.181623 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5212,"thread_start_us":5040,"threads_started":1}
I20260807 07:06:37.184419 17084 tablet_service.cc:1467] Processing CreateTablet for tablet 02fba9b150984fa1927fe756ffa1a9a5 (DEFAULT_TABLE table=test-table [id=80c3fa278b2445bfb38108fa7c2e6688]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:06:37.184810 17273 tablet_service.cc:1467] Processing CreateTablet for tablet 02fba9b150984fa1927fe756ffa1a9a5 (DEFAULT_TABLE table=test-table [id=80c3fa278b2445bfb38108fa7c2e6688]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:06:37.185247 16896 tablet_service.cc:1467] Processing CreateTablet for tablet 02fba9b150984fa1927fe756ffa1a9a5 (DEFAULT_TABLE table=test-table [id=80c3fa278b2445bfb38108fa7c2e6688]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:06:37.186129 17084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02fba9b150984fa1927fe756ffa1a9a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:06:37.186239 17273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02fba9b150984fa1927fe756ffa1a9a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:06:37.186553 16896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02fba9b150984fa1927fe756ffa1a9a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:06:37.198299 17405 tablet_bootstrap.cc:492] T 02fba9b150984fa1927fe756ffa1a9a5 P 36b9b64beecb47c5958dd5bf4058ff88: Bootstrap starting.
I20260807 07:06:37.201368 17408 tablet_bootstrap.cc:492] T 02fba9b150984fa1927fe756ffa1a9a5 P aa299af27c974ca4b661a728eee5b963: Bootstrap starting.
I20260807 07:06:37.202808 17405 tablet_bootstrap.cc:654] T 02fba9b150984fa1927fe756ffa1a9a5 P 36b9b64beecb47c5958dd5bf4058ff88: Neither blocks nor log segments found. Creating new log.
I20260807 07:06:37.204175 17405 log.cc:824] T 02fba9b150984fa1927fe756ffa1a9a5 P 36b9b64beecb47c5958dd5bf4058ff88: Log is configured to *not* fsync() on all Append() calls
I20260807 07:06:37.205677 17408 tablet_bootstrap.cc:654] T 02fba9b150984fa1927fe756ffa1a9a5 P aa299af27c974ca4b661a728eee5b963: Neither blocks nor log segments found. Creating new log.
I20260807 07:06:37.207036 17408 log.cc:824] T 02fba9b150984fa1927fe756ffa1a9a5 P aa299af27c974ca4b661a728eee5b963: Log is configured to *not* fsync() on all Append() calls
I20260807 07:06:37.207499 17407 tablet_bootstrap.cc:492] T 02fba9b150984fa1927fe756ffa1a9a5 P 8f80383906244d3c936b7e3da3f122b7: Bootstrap starting.
I20260807 07:06:37.210081 17408 tablet_bootstrap.cc:492] T 02fba9b150984fa1927fe756ffa1a9a5 P aa299af27c974ca4b661a728eee5b963: No bootstrap required, opened a new log
I20260807 07:06:37.210083 17405 tablet_bootstrap.cc:492] T 02fba9b150984fa1927fe756ffa1a9a5 P 36b9b64beecb47c5958dd5bf4058ff88: No bootstrap required, opened a new log
I20260807 07:06:37.210382 17405 ts_tablet_manager.cc:1397] T 02fba9b150984fa1927fe756ffa1a9a5 P 36b9b64beecb47c5958dd5bf4058ff88: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.004s
I20260807 07:06:37.210382 17408 ts_tablet_manager.cc:1397] T 02fba9b150984fa1927fe756ffa1a9a5 P aa299af27c974ca4b661a728eee5b963: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260807 07:06:37.211948 17407 tablet_bootstrap.cc:654] T 02fba9b150984fa1927fe756ffa1a9a5 P 8f80383906244d3c936b7e3da3f122b7: Neither blocks nor log segments found. Creating new log.
I20260807 07:06:37.213064 17407 log.cc:824] T 02fba9b150984fa1927fe756ffa1a9a5 P 8f80383906244d3c936b7e3da3f122b7: Log is configured to *not* fsync() on all Append() calls
I20260807 07:06:37.215765 17407 tablet_bootstrap.cc:492] T 02fba9b150984fa1927fe756ffa1a9a5 P 8f80383906244d3c936b7e3da3f122b7: No bootstrap required, opened a new log
I20260807 07:06:37.216022 17407 ts_tablet_manager.cc:1397] T 02fba9b150984fa1927fe756ffa1a9a5 P 8f80383906244d3c936b7e3da3f122b7: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260807 07:06:37.221220 17408 raft_consensus.cc:348] T 02fba9b150984fa1927fe756ffa1a9a5 P aa299af27c974ca4b661a728eee5b963 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa299af27c974ca4b661a728eee5b963" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 38337 } } peers { permanent_uuid: "8f80383906244d3c936b7e3da3f122b7" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 39811 } } peers { permanent_uuid: "36b9b64beecb47c5958dd5bf4058ff88" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 41585 } }
I20260807 07:06:37.221432 17405 raft_consensus.cc:348] T 02fba9b150984fa1927fe756ffa1a9a5 P 36b9b64beecb47c5958dd5bf4058ff88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa299af27c974ca4b661a728eee5b963" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 38337 } } peers { permanent_uuid: "8f80383906244d3c936b7e3da3f122b7" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 39811 } } peers { permanent_uuid: "36b9b64beecb47c5958dd5bf4058ff88" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 41585 } }
I20260807 07:06:37.221755 17408 raft_consensus.cc:374] T 02fba9b150984fa1927fe756ffa1a9a5 P aa299af27c974ca4b661a728eee5b963 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:06:37.221829 17405 raft_consensus.cc:374] T 02fba9b150984fa1927fe756ffa1a9a5 P 36b9b64beecb47c5958dd5bf4058ff88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:06:37.221855 17408 raft_consensus.cc:729] T 02fba9b150984fa1927fe756ffa1a9a5 P aa299af27c974ca4b661a728eee5b963 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa299af27c974ca4b661a728eee5b963, State: Initialized, Role: FOLLOWER
I20260807 07:06:37.221925 17405 raft_consensus.cc:729] T 02fba9b150984fa1927fe756ffa1a9a5 P 36b9b64beecb47c5958dd5bf4058ff88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36b9b64beecb47c5958dd5bf4058ff88, State: Initialized, Role: FOLLOWER
I20260807 07:06:37.222548 17408 consensus_queue.cc:260] T 02fba9b150984fa1927fe756ffa1a9a5 P aa299af27c974ca4b661a728eee5b963 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa299af27c974ca4b661a728eee5b963" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 38337 } } peers { permanent_uuid: "8f80383906244d3c936b7e3da3f122b7" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 39811 } } peers { permanent_uuid: "36b9b64beecb47c5958dd5bf4058ff88" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 41585 } }
I20260807 07:06:37.222548 17405 consensus_queue.cc:260] T 02fba9b150984fa1927fe756ffa1a9a5 P 36b9b64beecb47c5958dd5bf4058ff88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa299af27c974ca4b661a728eee5b963" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 38337 } } peers { permanent_uuid: "8f80383906244d3c936b7e3da3f122b7" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 39811 } } peers { permanent_uuid: "36b9b64beecb47c5958dd5bf4058ff88" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 41585 } }
I20260807 07:06:37.224177 17379 heartbeater.cc:503] Master 127.21.83.62:36201 was elected leader, sending a full tablet report...
I20260807 07:06:37.224516 17408 ts_tablet_manager.cc:1428] T 02fba9b150984fa1927fe756ffa1a9a5 P aa299af27c974ca4b661a728eee5b963: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260807 07:06:37.224298 17407 raft_consensus.cc:348] T 02fba9b150984fa1927fe756ffa1a9a5 P 8f80383906244d3c936b7e3da3f122b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa299af27c974ca4b661a728eee5b963" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 38337 } } peers { permanent_uuid: "8f80383906244d3c936b7e3da3f122b7" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 39811 } } peers { permanent_uuid: "36b9b64beecb47c5958dd5bf4058ff88" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 41585 } }
I20260807 07:06:37.224735 17407 raft_consensus.cc:374] T 02fba9b150984fa1927fe756ffa1a9a5 P 8f80383906244d3c936b7e3da3f122b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:06:37.224822 17407 raft_consensus.cc:729] T 02fba9b150984fa1927fe756ffa1a9a5 P 8f80383906244d3c936b7e3da3f122b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f80383906244d3c936b7e3da3f122b7, State: Initialized, Role: FOLLOWER
I20260807 07:06:37.225442 17407 consensus_queue.cc:260] T 02fba9b150984fa1927fe756ffa1a9a5 P 8f80383906244d3c936b7e3da3f122b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa299af27c974ca4b661a728eee5b963" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 38337 } } peers { permanent_uuid: "8f80383906244d3c936b7e3da3f122b7" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 39811 } } peers { permanent_uuid: "36b9b64beecb47c5958dd5bf4058ff88" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 41585 } }
I20260807 07:06:37.226086 17405 ts_tablet_manager.cc:1428] T 02fba9b150984fa1927fe756ffa1a9a5 P 36b9b64beecb47c5958dd5bf4058ff88: Time spent starting tablet: real 0.016s user 0.012s sys 0.003s
I20260807 07:06:37.226848 17407 ts_tablet_manager.cc:1428] T 02fba9b150984fa1927fe756ffa1a9a5 P 8f80383906244d3c936b7e3da3f122b7: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20260807 07:06:37.227484 17180 heartbeater.cc:503] Master 127.21.83.62:36201 was elected leader, sending a full tablet report...
W20260807 07:06:37.343405 17380 tablet.cc:2369] T 02fba9b150984fa1927fe756ffa1a9a5 P 36b9b64beecb47c5958dd5bf4058ff88: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 07:06:37.353163 17415 raft_consensus.cc:482] T 02fba9b150984fa1927fe756ffa1a9a5 P 8f80383906244d3c936b7e3da3f122b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:06:37.353503 17415 raft_consensus.cc:504] T 02fba9b150984fa1927fe756ffa1a9a5 P 8f80383906244d3c936b7e3da3f122b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa299af27c974ca4b661a728eee5b963" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 38337 } } peers { permanent_uuid: "8f80383906244d3c936b7e3da3f122b7" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 39811 } } peers { permanent_uuid: "36b9b64beecb47c5958dd5bf4058ff88" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 41585 } }
I20260807 07:06:37.355515 17415 leader_election.cc:288] T 02fba9b150984fa1927fe756ffa1a9a5 P 8f80383906244d3c936b7e3da3f122b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa299af27c974ca4b661a728eee5b963 (127.21.83.1:38337), 36b9b64beecb47c5958dd5bf4058ff88 (127.21.83.3:41585)
W20260807 07:06:37.365331 17181 tablet.cc:2369] T 02fba9b150984fa1927fe756ffa1a9a5 P 8f80383906244d3c936b7e3da3f122b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 07:06:37.365815 17403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:37.355948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:33597 (local address 127.21.83.3:41585)
0807 07:06:37.356319 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:37.356325 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:37.356857 (+ 532us) server_negotiation.cc:407] Connection header received
0807 07:06:37.357188 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:37.357194 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:37.357278 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:37.357386 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:06:37.358312 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:37.360019 (+ 1707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:37.363173 (+ 3154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:37.364479 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:37.364644 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:37.364769 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:37.365419 (+ 650us) server_negotiation.cc:299] Negotiation successful
0807 07:06:37.365576 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224}
I20260807 07:06:37.366197 17417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:37.355906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:38337 (local address 127.21.83.2:44455)
0807 07:06:37.356854 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:37.356870 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 07:06:37.357042 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:06:37.357462 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:06:37.357468 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:06:37.357514 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:06:37.358088 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:37.358106 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:37.360203 (+ 2097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:37.360208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:37.362872 (+ 2664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:37.362893 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:37.364786 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:37.364793 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:37.365116 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:37.365209 (+ 93us) client_negotiation.cc:769] Sending connection context
0807 07:06:37.365339 (+ 130us) client_negotiation.cc:240] Negotiation successful
0807 07:06:37.365547 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":841,"mutex_wait_us":310,"spinlock_wait_cycles":61568,"thread_start_us":605,"threads_started":1}
I20260807 07:06:37.366223 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:37.355855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.3:41585 (local address 127.21.83.2:33597)
0807 07:06:37.356735 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:37.356755 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 07:06:37.357013 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:06:37.357462 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:06:37.357468 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:06:37.357502 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:06:37.358124 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:37.358138 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:37.360202 (+ 2064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:37.360208 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:37.362879 (+ 2671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:37.362893 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:37.364793 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:37.364801 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:37.365119 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:37.365209 (+ 90us) client_negotiation.cc:769] Sending connection context
0807 07:06:37.365329 (+ 120us) client_negotiation.cc:240] Negotiation successful
0807 07:06:37.365693 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"mutex_wait_us":244,"thread_start_us":455,"threads_started":1}
I20260807 07:06:37.366673 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:37.356085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:44455 (local address 127.21.83.1:38337)
0807 07:06:37.356305 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:37.356313 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:37.356998 (+ 685us) server_negotiation.cc:407] Connection header received
0807 07:06:37.357187 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:37.357192 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:37.357290 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:37.357387 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:06:37.358312 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:37.360019 (+ 1707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:37.363173 (+ 3154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:37.364516 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:37.364648 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:37.364769 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:37.366091 (+ 1322us) server_negotiation.cc:299] Negotiation successful
0807 07:06:37.366250 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260807 07:06:37.367280 17307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02fba9b150984fa1927fe756ffa1a9a5" candidate_uuid: "8f80383906244d3c936b7e3da3f122b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36b9b64beecb47c5958dd5bf4058ff88" is_pre_election: true
I20260807 07:06:37.367509 16926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02fba9b150984fa1927fe756ffa1a9a5" candidate_uuid: "8f80383906244d3c936b7e3da3f122b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa299af27c974ca4b661a728eee5b963" is_pre_election: true
I20260807 07:06:37.367794 17307 raft_consensus.cc:2454] T 02fba9b150984fa1927fe756ffa1a9a5 P 36b9b64beecb47c5958dd5bf4058ff88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f80383906244d3c936b7e3da3f122b7 in term 0.
I20260807 07:06:37.368021 16926 raft_consensus.cc:2454] T 02fba9b150984fa1927fe756ffa1a9a5 P aa299af27c974ca4b661a728eee5b963 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f80383906244d3c936b7e3da3f122b7 in term 0.
I20260807 07:06:37.368462 17019 leader_election.cc:302] T 02fba9b150984fa1927fe756ffa1a9a5 P 8f80383906244d3c936b7e3da3f122b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36b9b64beecb47c5958dd5bf4058ff88, 8f80383906244d3c936b7e3da3f122b7; no voters:
I20260807 07:06:37.368935 17415 raft_consensus.cc:2781] T 02fba9b150984fa1927fe756ffa1a9a5 P 8f80383906244d3c936b7e3da3f122b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 07:06:37.369097 17415 raft_consensus.cc:482] T 02fba9b150984fa1927fe756ffa1a9a5 P 8f80383906244d3c936b7e3da3f122b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:06:37.369218 17415 raft_consensus.cc:3037] T 02fba9b150984fa1927fe756ffa1a9a5 P 8f80383906244d3c936b7e3da3f122b7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:06:37.371661 17415 raft_consensus.cc:504] T 02fba9b150984fa1927fe756ffa1a9a5 P 8f80383906244d3c936b7e3da3f122b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa299af27c974ca4b661a728eee5b963" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 38337 } } peers { permanent_uuid: "8f80383906244d3c936b7e3da3f122b7" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 39811 } } peers { permanent_uuid: "36b9b64beecb47c5958dd5bf4058ff88" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 41585 } }
I20260807 07:06:37.372287 17415 leader_election.cc:288] T 02fba9b150984fa1927fe756ffa1a9a5 P 8f80383906244d3c936b7e3da3f122b7 [CANDIDATE]: Term 1 election: Requested vote from peers aa299af27c974ca4b661a728eee5b963 (127.21.83.1:38337), 36b9b64beecb47c5958dd5bf4058ff88 (127.21.83.3:41585)
I20260807 07:06:37.372938 16926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02fba9b150984fa1927fe756ffa1a9a5" candidate_uuid: "8f80383906244d3c936b7e3da3f122b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa299af27c974ca4b661a728eee5b963"
I20260807 07:06:37.372980 17307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02fba9b150984fa1927fe756ffa1a9a5" candidate_uuid: "8f80383906244d3c936b7e3da3f122b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36b9b64beecb47c5958dd5bf4058ff88"
I20260807 07:06:37.373158 16926 raft_consensus.cc:3037] T 02fba9b150984fa1927fe756ffa1a9a5 P aa299af27c974ca4b661a728eee5b963 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:06:37.373224 17307 raft_consensus.cc:3037] T 02fba9b150984fa1927fe756ffa1a9a5 P 36b9b64beecb47c5958dd5bf4058ff88 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:06:37.375530 16926 raft_consensus.cc:2454] T 02fba9b150984fa1927fe756ffa1a9a5 P aa299af27c974ca4b661a728eee5b963 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f80383906244d3c936b7e3da3f122b7 in term 1.
I20260807 07:06:37.375545 17307 raft_consensus.cc:2454] T 02fba9b150984fa1927fe756ffa1a9a5 P 36b9b64beecb47c5958dd5bf4058ff88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f80383906244d3c936b7e3da3f122b7 in term 1.
I20260807 07:06:37.376072 17021 leader_election.cc:302] T 02fba9b150984fa1927fe756ffa1a9a5 P 8f80383906244d3c936b7e3da3f122b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f80383906244d3c936b7e3da3f122b7, aa299af27c974ca4b661a728eee5b963; no voters:
I20260807 07:06:37.376437 17415 raft_consensus.cc:2781] T 02fba9b150984fa1927fe756ffa1a9a5 P 8f80383906244d3c936b7e3da3f122b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:06:37.376982 17415 raft_consensus.cc:686] T 02fba9b150984fa1927fe756ffa1a9a5 P 8f80383906244d3c936b7e3da3f122b7 [term 1 LEADER]: Becoming Leader. State: Replica: 8f80383906244d3c936b7e3da3f122b7, State: Running, Role: LEADER
I20260807 07:06:37.377537 17415 consensus_queue.cc:237] T 02fba9b150984fa1927fe756ffa1a9a5 P 8f80383906244d3c936b7e3da3f122b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa299af27c974ca4b661a728eee5b963" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 38337 } } peers { permanent_uuid: "8f80383906244d3c936b7e3da3f122b7" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 39811 } } peers { permanent_uuid: "36b9b64beecb47c5958dd5bf4058ff88" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 41585 } }
I20260807 07:06:37.382246 16719 catalog_manager.cc:5495] T 02fba9b150984fa1927fe756ffa1a9a5 P 8f80383906244d3c936b7e3da3f122b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f80383906244d3c936b7e3da3f122b7 (127.21.83.2). New cstate: current_term: 1 leader_uuid: "8f80383906244d3c936b7e3da3f122b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa299af27c974ca4b661a728eee5b963" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 38337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f80383906244d3c936b7e3da3f122b7" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 39811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36b9b64beecb47c5958dd5bf4058ff88" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 41585 } health_report { overall_health: UNKNOWN } } }
W20260807 07:06:37.403390 16995 tablet.cc:2369] T 02fba9b150984fa1927fe756ffa1a9a5 P aa299af27c974ca4b661a728eee5b963: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 07:06:37.407747 17402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:37.402449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56566 (local address 127.21.83.2:39811)
0807 07:06:37.402655 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:37.402660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:37.402680 (+ 20us) server_negotiation.cc:407] Connection header received
0807 07:06:37.402765 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:37.402768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:37.402879 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:37.403003 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:06:37.403428 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:37.404169 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:37.405629 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:37.406061 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:37.406161 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:37.406618 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:06:37.407146 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:06:37.407278 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:37.407423 (+ 145us) server_negotiation.cc:299] Negotiation successful
0807 07:06:37.407469 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 07:06:37.416283 16926 raft_consensus.cc:1261] T 02fba9b150984fa1927fe756ffa1a9a5 P aa299af27c974ca4b661a728eee5b963 [term 1 FOLLOWER]: Refusing update from remote peer 8f80383906244d3c936b7e3da3f122b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:06:37.416283 17307 raft_consensus.cc:1261] T 02fba9b150984fa1927fe756ffa1a9a5 P 36b9b64beecb47c5958dd5bf4058ff88 [term 1 FOLLOWER]: Refusing update from remote peer 8f80383906244d3c936b7e3da3f122b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:06:37.417485 17418 consensus_queue.cc:1035] T 02fba9b150984fa1927fe756ffa1a9a5 P 8f80383906244d3c936b7e3da3f122b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36b9b64beecb47c5958dd5bf4058ff88" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 41585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:06:37.417869 17415 consensus_queue.cc:1035] T 02fba9b150984fa1927fe756ffa1a9a5 P 8f80383906244d3c936b7e3da3f122b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa299af27c974ca4b661a728eee5b963" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 38337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:06:37.430698 17424 mvcc.cc:205] Tried to move back new op lower bound from 7315809883806040064 to 7315809883662163968. Current Snapshot: MvccSnapshot[applied={T|T < 7315809883806040064}]
I20260807 07:06:37.443007 16717 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:06:37.443333 16717 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:06:37.447760 16717 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1173183a0b3e48c48e1b06a3143f5e5a: Sending DeleteTablet for 3 replicas of tablet 02fba9b150984fa1927fe756ffa1a9a5
I20260807 07:06:37.448953 16896 tablet_service.cc:1514] Processing DeleteTablet for tablet 02fba9b150984fa1927fe756ffa1a9a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:06:37 UTC) from {username='slave'} at 127.0.0.1:39752
I20260807 07:06:37.449779 17273 tablet_service.cc:1514] Processing DeleteTablet for tablet 02fba9b150984fa1927fe756ffa1a9a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:06:37 UTC) from {username='slave'} at 127.0.0.1:53022
I20260807 07:06:37.450008 17437 tablet_replica.cc:330] T 02fba9b150984fa1927fe756ffa1a9a5 P aa299af27c974ca4b661a728eee5b963: stopping tablet replica
I20260807 07:06:37.450445 17437 raft_consensus.cc:2229] T 02fba9b150984fa1927fe756ffa1a9a5 P aa299af27c974ca4b661a728eee5b963 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:06:37.451022 17437 raft_consensus.cc:2258] T 02fba9b150984fa1927fe756ffa1a9a5 P aa299af27c974ca4b661a728eee5b963 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:06:37.451833 17438 tablet_replica.cc:330] T 02fba9b150984fa1927fe756ffa1a9a5 P 36b9b64beecb47c5958dd5bf4058ff88: stopping tablet replica
I20260807 07:06:37.452145 17438 raft_consensus.cc:2229] T 02fba9b150984fa1927fe756ffa1a9a5 P 36b9b64beecb47c5958dd5bf4058ff88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:06:37.452389 17084 tablet_service.cc:1514] Processing DeleteTablet for tablet 02fba9b150984fa1927fe756ffa1a9a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:06:37 UTC) from {username='slave'} at 127.0.0.1:56564
I20260807 07:06:37.452523 17438 raft_consensus.cc:2258] T 02fba9b150984fa1927fe756ffa1a9a5 P 36b9b64beecb47c5958dd5bf4058ff88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:06:37.453495 17439 tablet_replica.cc:330] T 02fba9b150984fa1927fe756ffa1a9a5 P 8f80383906244d3c936b7e3da3f122b7: stopping tablet replica
I20260807 07:06:37.454036 17439 raft_consensus.cc:2229] T 02fba9b150984fa1927fe756ffa1a9a5 P 8f80383906244d3c936b7e3da3f122b7 [term 1 LEADER]: Raft consensus shutting down.
I20260807 07:06:37.454842 17439 raft_consensus.cc:2258] T 02fba9b150984fa1927fe756ffa1a9a5 P 8f80383906244d3c936b7e3da3f122b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:06:37.454880 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 16801
I20260807 07:06:37.455085 17437 ts_tablet_manager.cc:1905] T 02fba9b150984fa1927fe756ffa1a9a5 P aa299af27c974ca4b661a728eee5b963: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:06:37.460280 17438 ts_tablet_manager.cc:1905] T 02fba9b150984fa1927fe756ffa1a9a5 P 36b9b64beecb47c5958dd5bf4058ff88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:06:37.460923 17439 ts_tablet_manager.cc:1905] T 02fba9b150984fa1927fe756ffa1a9a5 P 8f80383906244d3c936b7e3da3f122b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:06:37.462697 17437 ts_tablet_manager.cc:1918] T 02fba9b150984fa1927fe756ffa1a9a5 P aa299af27c974ca4b661a728eee5b963: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:06:37.462877 17437 log.cc:1196] T 02fba9b150984fa1927fe756ffa1a9a5 P aa299af27c974ca4b661a728eee5b963: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-0/wal/wals/02fba9b150984fa1927fe756ffa1a9a5
I20260807 07:06:37.463421 17437 ts_tablet_manager.cc:1939] T 02fba9b150984fa1927fe756ffa1a9a5 P aa299af27c974ca4b661a728eee5b963: Deleting consensus metadata
I20260807 07:06:37.465523 16700 catalog_manager.cc:4841] TS aa299af27c974ca4b661a728eee5b963 (127.21.83.1:38337): tablet 02fba9b150984fa1927fe756ffa1a9a5 (table test-table [id=80c3fa278b2445bfb38108fa7c2e6688]) successfully deleted
I20260807 07:06:37.469551 17438 ts_tablet_manager.cc:1918] T 02fba9b150984fa1927fe756ffa1a9a5 P 36b9b64beecb47c5958dd5bf4058ff88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:06:37.469777 17438 log.cc:1196] T 02fba9b150984fa1927fe756ffa1a9a5 P 36b9b64beecb47c5958dd5bf4058ff88: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-2/wal/wals/02fba9b150984fa1927fe756ffa1a9a5
I20260807 07:06:37.470005 17439 ts_tablet_manager.cc:1918] T 02fba9b150984fa1927fe756ffa1a9a5 P 8f80383906244d3c936b7e3da3f122b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:06:37.470263 17439 log.cc:1196] T 02fba9b150984fa1927fe756ffa1a9a5 P 8f80383906244d3c936b7e3da3f122b7: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786086078080656-21836-0/minicluster-data/ts-1/wal/wals/02fba9b150984fa1927fe756ffa1a9a5
I20260807 07:06:37.470367 17438 ts_tablet_manager.cc:1939] T 02fba9b150984fa1927fe756ffa1a9a5 P 36b9b64beecb47c5958dd5bf4058ff88: Deleting consensus metadata
I20260807 07:06:37.471014 17439 ts_tablet_manager.cc:1939] T 02fba9b150984fa1927fe756ffa1a9a5 P 8f80383906244d3c936b7e3da3f122b7: Deleting consensus metadata
I20260807 07:06:37.472759 16696 catalog_manager.cc:4841] TS 36b9b64beecb47c5958dd5bf4058ff88 (127.21.83.3:41585): tablet 02fba9b150984fa1927fe756ffa1a9a5 (table test-table [id=80c3fa278b2445bfb38108fa7c2e6688]) successfully deleted
I20260807 07:06:37.473129 16696 catalog_manager.cc:4841] TS 8f80383906244d3c936b7e3da3f122b7 (127.21.83.2:39811): tablet 02fba9b150984fa1927fe756ffa1a9a5 (table test-table [id=80c3fa278b2445bfb38108fa7c2e6688]) successfully deleted
I20260807 07:06:37.474993 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:37.456073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39758 (local address 127.21.83.1:38337)
0807 07:06:37.456257 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:37.456263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:37.466376 (+ 10113us) server_negotiation.cc:407] Connection header received
0807 07:06:37.466669 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:37.466675 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:37.466788 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:37.466865 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:37.467755 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:37.468862 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:37.470817 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:37.471407 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:37.471526 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:37.473155 (+ 1629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:37.473168 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:37.473189 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:06:37.474022 (+ 833us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:37.474104 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:06:37.474143 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:37.474267 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:37.474473 (+ 206us) server_negotiation.cc:299] Negotiation successful
0807 07:06:37.474607 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 07:06:37.475682 16987 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:06:37.566504 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 16801
I20260807 07:06:37.596050 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 16999
I20260807 07:06:37.603298 17402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:37.596947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56574 (local address 127.21.83.2:39811)
0807 07:06:37.597165 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:37.597171 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:37.597191 (+ 20us) server_negotiation.cc:407] Connection header received
0807 07:06:37.597248 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:37.597252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:37.597333 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:37.597417 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:37.597831 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:37.598656 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:37.599920 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:37.600360 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:37.600479 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:37.601676 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:37.601686 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:37.601704 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:06:37.602490 (+ 786us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:37.602563 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:06:37.602603 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:37.602703 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:37.602808 (+ 105us) server_negotiation.cc:299] Negotiation successful
0807 07:06:37.602912 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 07:06:37.604156 17175 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:06:37.713735 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 16999
I20260807 07:06:37.738705 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 17184
I20260807 07:06:37.744696 17403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:37.739396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53028 (local address 127.21.83.3:41585)
0807 07:06:37.739553 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:37.739557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:37.739596 (+ 39us) server_negotiation.cc:407] Connection header received
0807 07:06:37.739747 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:37.739751 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:37.739839 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:37.739922 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:37.740261 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:37.741028 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:37.742275 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:37.742650 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:37.742733 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:37.743608 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:37.743616 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:37.743629 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:06:37.744147 (+ 518us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:37.744193 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:06:37.744218 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:37.744299 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:37.744366 (+ 67us) server_negotiation.cc:299] Negotiation successful
0807 07:06:37.744448 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260807 07:06:37.745388 17370 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:06:37.833412 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 17184
I20260807 07:06:37.855158 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 16669
I20260807 07:06:37.856396 16762 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:06:37.967078 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 16669
2026-08-07T07:06:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2945 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 07:06:38 dist-test-slave-ncp9 krb5kdc[17450](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 07:06:38 dist-test-slave-ncp9 krb5kdc[17450](info): set up 2 sockets
Aug 07 07:06:38 dist-test-slave-ncp9 krb5kdc[17450](info): commencing operation
krb5kdc: starting...
W20260807 07:06:39.201242 21836 mini_kdc.cc:121] Time spent starting KDC: real 1.186s user 0.000s sys 0.057s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Aug 07 07:06:39 dist-test-slave-ncp9 krb5kdc[17450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T07:06:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:06:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
I20260807 07:06:39.557825 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:34935
--webserver_interface=127.21.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:46013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:34935
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:06:39.909726 17468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:06:39.910030 17468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:06:39.910159 17468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:06:39.917665 17468 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:06:39.917774 17468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:06:39.917832 17468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:06:39.917913 17468 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:06:39.917976 17468 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:06:39.928083 17468 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:46013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:34935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:34935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:06:39.929472 17468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:06:39.930817 17468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:06:39.942570 17473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:39.942543 17478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:39.942660 17476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:06:39.943665 17468 server_base.cc:1034] running on GCE node
I20260807 07:06:39.944483 17468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:06:39.946019 17468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:06:39.947235 17468 hybrid_clock.cc:648] HybridClock initialized: now 1786086399947187 us; error 41 us; skew 500 ppm
I20260807 07:06:39.947606 17468 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:06:39 dist-test-slave-ncp9 krb5kdc[17450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086399, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:06:39.950960 17468 init.cc:377] Logged in from keytab as kudu/127.21.83.62@KRBTEST.COM (short username kudu)
I20260807 07:06:39.952837 17468 webserver.cc:460] Webserver started at http://127.21.83.62:34115/ using document root <none> and password file <none>
I20260807 07:06:39.953655 17468 fs_manager.cc:359] Metadata directory not provided
I20260807 07:06:39.953794 17468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:06:39.954121 17468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:06:39.957521 17468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "253f3a8ce6bd4ff3946e29a181278b9d"
format_stamp: "Formatted at 2026-08-07 07:06:39 on dist-test-slave-ncp9"
server_key: "855db67351e7e84893b3ae092b372cb3"
server_key_iv: "b7ec389f21114b2361218c23018bf3be"
server_key_version: "encryptionkey@0"
I20260807 07:06:39.958405 17468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "253f3a8ce6bd4ff3946e29a181278b9d"
format_stamp: "Formatted at 2026-08-07 07:06:39 on dist-test-slave-ncp9"
server_key: "855db67351e7e84893b3ae092b372cb3"
server_key_iv: "b7ec389f21114b2361218c23018bf3be"
server_key_version: "encryptionkey@0"
I20260807 07:06:39.964473 17468 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260807 07:06:39.969398 17487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:39.971124 17468 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260807 07:06:39.971333 17468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "253f3a8ce6bd4ff3946e29a181278b9d"
format_stamp: "Formatted at 2026-08-07 07:06:39 on dist-test-slave-ncp9"
server_key: "855db67351e7e84893b3ae092b372cb3"
server_key_iv: "b7ec389f21114b2361218c23018bf3be"
server_key_version: "encryptionkey@0"
I20260807 07:06:39.971554 17468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:06:39.989410 17468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:06:39.993925 17468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:06:39.994231 17468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:06:40.013950 17468 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.62:34935
I20260807 07:06:40.014034 17563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.62:34935 every 8 connection(s)
I20260807 07:06:40.015583 17468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:06:40.019523 17565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:06:40.024645 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 17468
I20260807 07:06:40.024866 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:06:40.025418 21836 external_mini_cluster.cc:1428] Setting key af779c597bcdc262b9998423011d0699
I20260807 07:06:40.031394 17565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 253f3a8ce6bd4ff3946e29a181278b9d: Bootstrap starting.
I20260807 07:06:40.035244 17565 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 253f3a8ce6bd4ff3946e29a181278b9d: Neither blocks nor log segments found. Creating new log.
I20260807 07:06:40.036517 17565 log.cc:824] T 00000000000000000000000000000000 P 253f3a8ce6bd4ff3946e29a181278b9d: Log is configured to *not* fsync() on all Append() calls
I20260807 07:06:40.039391 17565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 253f3a8ce6bd4ff3946e29a181278b9d: No bootstrap required, opened a new log
Aug 07 07:06:40 dist-test-slave-ncp9 krb5kdc[17450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:06:40.048702 17565 raft_consensus.cc:348] T 00000000000000000000000000000000 P 253f3a8ce6bd4ff3946e29a181278b9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "253f3a8ce6bd4ff3946e29a181278b9d" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 34935 } }
I20260807 07:06:40.049146 17565 raft_consensus.cc:374] T 00000000000000000000000000000000 P 253f3a8ce6bd4ff3946e29a181278b9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:06:40.049264 17565 raft_consensus.cc:729] T 00000000000000000000000000000000 P 253f3a8ce6bd4ff3946e29a181278b9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 253f3a8ce6bd4ff3946e29a181278b9d, State: Initialized, Role: FOLLOWER
I20260807 07:06:40.049641 17568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:40.027371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45650 (local address 127.21.83.62:34935)
0807 07:06:40.028248 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:40.028260 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:40.028304 (+ 44us) server_negotiation.cc:407] Connection header received
0807 07:06:40.030175 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:40.030199 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:40.030622 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:40.030946 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:40.032995 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:40.034355 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:40.036070 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:40.036828 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:40.037028 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:40.041567 (+ 4539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:40.041576 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:40.041592 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:06:40.041636 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:40.045095 (+ 3459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:40.045996 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:40.046003 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:40.046021 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:40.046197 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:40.046622 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:40.046625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:40.046627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:40.047275 (+ 648us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:40.047507 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:40.047998 (+ 491us) server_negotiation.cc:299] Negotiation successful
0807 07:06:40.048402 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":426,"threads_started":1}
I20260807 07:06:40.049978 17565 consensus_queue.cc:260] T 00000000000000000000000000000000 P 253f3a8ce6bd4ff3946e29a181278b9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "253f3a8ce6bd4ff3946e29a181278b9d" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 34935 } }
I20260807 07:06:40.050259 17565 raft_consensus.cc:388] T 00000000000000000000000000000000 P 253f3a8ce6bd4ff3946e29a181278b9d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:06:40.050395 17565 raft_consensus.cc:482] T 00000000000000000000000000000000 P 253f3a8ce6bd4ff3946e29a181278b9d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:06:40.050562 17565 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 253f3a8ce6bd4ff3946e29a181278b9d [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:06:40.053741 17565 raft_consensus.cc:504] T 00000000000000000000000000000000 P 253f3a8ce6bd4ff3946e29a181278b9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "253f3a8ce6bd4ff3946e29a181278b9d" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 34935 } }
I20260807 07:06:40.054320 17565 leader_election.cc:302] T 00000000000000000000000000000000 P 253f3a8ce6bd4ff3946e29a181278b9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 253f3a8ce6bd4ff3946e29a181278b9d; no voters:
I20260807 07:06:40.055083 17565 leader_election.cc:288] T 00000000000000000000000000000000 P 253f3a8ce6bd4ff3946e29a181278b9d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 07:06:40.055245 17571 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 253f3a8ce6bd4ff3946e29a181278b9d [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:06:40.056001 17571 raft_consensus.cc:686] T 00000000000000000000000000000000 P 253f3a8ce6bd4ff3946e29a181278b9d [term 1 LEADER]: Becoming Leader. State: Replica: 253f3a8ce6bd4ff3946e29a181278b9d, State: Running, Role: LEADER
I20260807 07:06:40.056574 17571 consensus_queue.cc:237] T 00000000000000000000000000000000 P 253f3a8ce6bd4ff3946e29a181278b9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "253f3a8ce6bd4ff3946e29a181278b9d" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 34935 } }
I20260807 07:06:40.057284 17565 sys_catalog.cc:564] T 00000000000000000000000000000000 P 253f3a8ce6bd4ff3946e29a181278b9d [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:06:40.059347 17572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 253f3a8ce6bd4ff3946e29a181278b9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "253f3a8ce6bd4ff3946e29a181278b9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "253f3a8ce6bd4ff3946e29a181278b9d" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 34935 } } }
I20260807 07:06:40.059441 17573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 253f3a8ce6bd4ff3946e29a181278b9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 253f3a8ce6bd4ff3946e29a181278b9d. Latest consensus state: current_term: 1 leader_uuid: "253f3a8ce6bd4ff3946e29a181278b9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "253f3a8ce6bd4ff3946e29a181278b9d" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 34935 } } }
I20260807 07:06:40.060171 17573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 253f3a8ce6bd4ff3946e29a181278b9d [sys.catalog]: This master's current role is: LEADER
I20260807 07:06:40.060738 17572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 253f3a8ce6bd4ff3946e29a181278b9d [sys.catalog]: This master's current role is: LEADER
I20260807 07:06:40.061923 17577 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:06:40.069087 17577 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:06:40.078737 17577 catalog_manager.cc:1347] Generated new cluster ID: e6c478f0188b469793ac06edb18d514d
I20260807 07:06:40.078893 17577 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:06:40.104140 17577 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:06:40.106091 17577 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:06:40.115605 17577 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 253f3a8ce6bd4ff3946e29a181278b9d: Generated new TSK 0
I20260807 07:06:40.116752 17577 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.83.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab.
Entry for principal kudu/127.21.83.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab.
I20260807 07:06:40.200225 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.1:0
--local_ip_for_outbound_sockets=127.21.83.1
--webserver_interface=127.21.83.1
--webserver_port=0
--tserver_master_addrs=127.21.83.62:34935
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab
--principal=kudu/127.21.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:46013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:06:40.553864 17599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:06:40.554174 17599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:06:40.554330 17599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:06:40.561668 17599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:06:40.561844 17599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:06:40.562050 17599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.1
I20260807 07:06:40.572587 17599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:46013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.1.keytab
--principal=kudu/127.21.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.83.1
--webserver_port=0
--tserver_master_addrs=127.21.83.62:34935
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.1
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:06:40.574007 17599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:06:40.575500 17599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:06:40.588696 17604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:40.588795 17605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:40.588827 17607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:06:40.590230 17599 server_base.cc:1034] running on GCE node
I20260807 07:06:40.590860 17599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:06:40.591785 17599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:06:40.592983 17599 hybrid_clock.cc:648] HybridClock initialized: now 1786086400592939 us; error 38 us; skew 500 ppm
I20260807 07:06:40.593291 17599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:06:40 dist-test-slave-ncp9 krb5kdc[17450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086400, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:06:40.596340 17599 init.cc:377] Logged in from keytab as kudu/127.21.83.1@KRBTEST.COM (short username kudu)
I20260807 07:06:40.597800 17599 webserver.cc:460] Webserver started at http://127.21.83.1:46483/ using document root <none> and password file <none>
I20260807 07:06:40.598453 17599 fs_manager.cc:359] Metadata directory not provided
I20260807 07:06:40.598564 17599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:06:40.598861 17599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:06:40.601701 17599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-0/data/instance:
uuid: "f04d64dce74e4cf5a73f023cfa711d51"
format_stamp: "Formatted at 2026-08-07 07:06:40 on dist-test-slave-ncp9"
server_key: "76d76eea27061a797cdfc4209ddca557"
server_key_iv: "1ae3f4594dd2d09523b8a721bf6a2ed1"
server_key_version: "encryptionkey@0"
I20260807 07:06:40.602445 17599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-0/wal/instance:
uuid: "f04d64dce74e4cf5a73f023cfa711d51"
format_stamp: "Formatted at 2026-08-07 07:06:40 on dist-test-slave-ncp9"
server_key: "76d76eea27061a797cdfc4209ddca557"
server_key_iv: "1ae3f4594dd2d09523b8a721bf6a2ed1"
server_key_version: "encryptionkey@0"
I20260807 07:06:40.607481 17599 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 07:06:40.610839 17615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:40.612030 17599 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 07:06:40.612185 17599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-0/wal
uuid: "f04d64dce74e4cf5a73f023cfa711d51"
format_stamp: "Formatted at 2026-08-07 07:06:40 on dist-test-slave-ncp9"
server_key: "76d76eea27061a797cdfc4209ddca557"
server_key_iv: "1ae3f4594dd2d09523b8a721bf6a2ed1"
server_key_version: "encryptionkey@0"
I20260807 07:06:40.612372 17599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:06:40.625253 17599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:06:40.629534 17599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:06:40.629806 17599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:06:40.630776 17599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:06:40.632153 17599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:06:40.632234 17599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:40.632359 17599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:06:40.632413 17599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:40.664520 17599 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.1:41367
I20260807 07:06:40.664541 17788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.1:41367 every 8 connection(s)
I20260807 07:06:40.665946 17599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
I20260807 07:06:40.676352 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 17599
I20260807 07:06:40.676530 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-0/wal/instance
I20260807 07:06:40.677165 21836 external_mini_cluster.cc:1428] Setting key 5cfd44c00d2c305356f5ee0ab7f68f7d
Aug 07 07:06:40 dist-test-slave-ncp9 krb5kdc[17450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086400, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.1@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:06:40.686834 17792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:40.668670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:43621 (local address 127.21.83.62:34935)
0807 07:06:40.669145 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:40.669150 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:40.670203 (+ 1053us) server_negotiation.cc:407] Connection header received
0807 07:06:40.672140 (+ 1937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:40.672146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:40.672234 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:40.672337 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:40.674112 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:40.674932 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:40.676610 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:40.677266 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:40.677416 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:40.681443 (+ 4027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:40.681448 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:40.681451 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:06:40.681484 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:40.683863 (+ 2379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:40.684666 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:40.684670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:40.684672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:40.684785 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:40.685245 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:40.685248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:40.685250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:40.685501 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:40.685618 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:40.686401 (+ 783us) server_negotiation.cc:299] Negotiation successful
0807 07:06:40.686573 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":254,"threads_started":1}
I20260807 07:06:40.688131 17793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:40.669182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:34935 (local address 127.21.83.1:43621)
0807 07:06:40.669997 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:40.670043 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 07:06:40.671822 (+ 1779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:06:40.672628 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:06:40.672638 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:06:40.673296 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:06:40.673917 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:40.673936 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:40.675121 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:40.675125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:40.676449 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:40.676463 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:40.677497 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:40.677502 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:40.677709 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:40.678476 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:06:40.678512 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:06:40.681188 (+ 2676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:06:40.684000 (+ 2812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:06:40.684009 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:06:40.684025 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:06:40.684563 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:06:40.684874 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:06:40.684878 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:06:40.684881 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:06:40.685151 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:06:40.685617 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:06:40.685623 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:06:40.686032 (+ 409us) client_negotiation.cc:769] Sending connection context
0807 07:06:40.686334 (+ 302us) client_negotiation.cc:240] Negotiation successful
0807 07:06:40.686845 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":338,"threads_started":1}
I20260807 07:06:40.689718 17790 heartbeater.cc:347] Connected to a master server at 127.21.83.62:34935
I20260807 07:06:40.690021 17790 heartbeater.cc:464] Registering TS with master...
I20260807 07:06:40.690723 17790 heartbeater.cc:511] Master 127.21.83.62:34935 requested a full tablet report, sending...
I20260807 07:06:40.692689 17509 ts_manager.cc:194] Registered new tserver with Master: f04d64dce74e4cf5a73f023cfa711d51 (127.21.83.1:41367)
I20260807 07:06:40.694137 17509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.1@KRBTEST.COM'} at 127.21.83.1:43621
WARNING: no policy specified for kudu/127.21.83.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab.
Entry for principal kudu/127.21.83.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab.
I20260807 07:06:40.751397 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.2:0
--local_ip_for_outbound_sockets=127.21.83.2
--webserver_interface=127.21.83.2
--webserver_port=0
--tserver_master_addrs=127.21.83.62:34935
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab
--principal=kudu/127.21.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:46013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:06:41.103055 17796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:06:41.103379 17796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:06:41.103510 17796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:06:41.111017 17796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:06:41.111136 17796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:06:41.111297 17796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.2
I20260807 07:06:41.121479 17796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:46013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.2.keytab
--principal=kudu/127.21.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.83.2
--webserver_port=0
--tserver_master_addrs=127.21.83.62:34935
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.2
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:06:41.122783 17796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:06:41.124053 17796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:06:41.136492 17807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:41.136637 17805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:41.136543 17810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:06:41.137662 17796 server_base.cc:1034] running on GCE node
I20260807 07:06:41.138285 17796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:06:41.139212 17796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:06:41.140421 17796 hybrid_clock.cc:648] HybridClock initialized: now 1786086401140358 us; error 50 us; skew 500 ppm
I20260807 07:06:41.140738 17796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:06:41 dist-test-slave-ncp9 krb5kdc[17450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086401, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:06:41.143769 17796 init.cc:377] Logged in from keytab as kudu/127.21.83.2@KRBTEST.COM (short username kudu)
I20260807 07:06:41.145202 17796 webserver.cc:460] Webserver started at http://127.21.83.2:39593/ using document root <none> and password file <none>
I20260807 07:06:41.145900 17796 fs_manager.cc:359] Metadata directory not provided
I20260807 07:06:41.146028 17796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:06:41.146338 17796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:06:41.149293 17796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-1/data/instance:
uuid: "1533cea80c9f4aef9cf7f9d06f90a0b4"
format_stamp: "Formatted at 2026-08-07 07:06:41 on dist-test-slave-ncp9"
server_key: "45a59b883da54589da8b8423ec1cd37c"
server_key_iv: "3cb0b3cafc3783b047ff306d15d56668"
server_key_version: "encryptionkey@0"
I20260807 07:06:41.150049 17796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-1/wal/instance:
uuid: "1533cea80c9f4aef9cf7f9d06f90a0b4"
format_stamp: "Formatted at 2026-08-07 07:06:41 on dist-test-slave-ncp9"
server_key: "45a59b883da54589da8b8423ec1cd37c"
server_key_iv: "3cb0b3cafc3783b047ff306d15d56668"
server_key_version: "encryptionkey@0"
I20260807 07:06:41.155124 17796 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 07:06:41.158667 17818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:41.159961 17796 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 07:06:41.160115 17796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-1/wal
uuid: "1533cea80c9f4aef9cf7f9d06f90a0b4"
format_stamp: "Formatted at 2026-08-07 07:06:41 on dist-test-slave-ncp9"
server_key: "45a59b883da54589da8b8423ec1cd37c"
server_key_iv: "3cb0b3cafc3783b047ff306d15d56668"
server_key_version: "encryptionkey@0"
I20260807 07:06:41.160310 17796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:06:41.181941 17796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:06:41.186607 17796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:06:41.186897 17796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:06:41.187896 17796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:06:41.189412 17796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:06:41.189494 17796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:41.189608 17796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:06:41.189661 17796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:41.221377 17796 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.2:32993
I20260807 07:06:41.221426 17987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.2:32993 every 8 connection(s)
I20260807 07:06:41.222673 17796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
I20260807 07:06:41.227439 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 17796
I20260807 07:06:41.227609 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-1/wal/instance
I20260807 07:06:41.228024 21836 external_mini_cluster.cc:1428] Setting key 6f8fb1a2178f6fa3f0a1ae09c636f956
Aug 07 07:06:41 dist-test-slave-ncp9 krb5kdc[17450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086401, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.2@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:06:41.243587 17991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:41.225129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:44467 (local address 127.21.83.62:34935)
0807 07:06:41.225580 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:41.225585 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:41.226484 (+ 899us) server_negotiation.cc:407] Connection header received
0807 07:06:41.228403 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:41.228409 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:41.228494 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:41.228606 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:41.230358 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:41.231096 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:41.233213 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:41.233641 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:41.233745 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:41.237811 (+ 4066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:41.237816 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:41.237820 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:06:41.237856 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:41.240509 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:41.241383 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:41.241387 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:41.241389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:41.241524 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:41.241956 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:41.241958 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:41.241960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:41.242234 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:41.242343 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:41.243131 (+ 788us) server_negotiation.cc:299] Negotiation successful
0807 07:06:41.243313 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":222,"threads_started":1}
I20260807 07:06:41.244844 17992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:41.225642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:34935 (local address 127.21.83.2:44467)
0807 07:06:41.226337 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:41.226373 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0807 07:06:41.228150 (+ 1777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:06:41.228883 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:06:41.228913 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:06:41.229470 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:06:41.230170 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:41.230199 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:41.231257 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:41.231263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:41.233042 (+ 1779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:41.233057 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:41.233858 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:41.233865 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:41.234139 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:41.234881 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:06:41.234914 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:06:41.237545 (+ 2631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:06:41.240661 (+ 3116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:06:41.240669 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:06:41.240687 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:06:41.241278 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:06:41.241624 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:06:41.241628 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:06:41.241631 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:06:41.241865 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:06:41.242348 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:06:41.242356 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:06:41.242765 (+ 409us) client_negotiation.cc:769] Sending connection context
0807 07:06:41.243052 (+ 287us) client_negotiation.cc:240] Negotiation successful
0807 07:06:41.243566 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":276,"threads_started":1}
I20260807 07:06:41.246454 17988 heartbeater.cc:347] Connected to a master server at 127.21.83.62:34935
I20260807 07:06:41.246744 17988 heartbeater.cc:464] Registering TS with master...
I20260807 07:06:41.247463 17988 heartbeater.cc:511] Master 127.21.83.62:34935 requested a full tablet report, sending...
I20260807 07:06:41.249042 17509 ts_manager.cc:194] Registered new tserver with Master: 1533cea80c9f4aef9cf7f9d06f90a0b4 (127.21.83.2:32993)
I20260807 07:06:41.249861 17509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.2@KRBTEST.COM'} at 127.21.83.2:44467
WARNING: no policy specified for kudu/127.21.83.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab.
Entry for principal kudu/127.21.83.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab.
I20260807 07:06:41.304598 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.3:0
--local_ip_for_outbound_sockets=127.21.83.3
--webserver_interface=127.21.83.3
--webserver_port=0
--tserver_master_addrs=127.21.83.62:34935
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab
--principal=kudu/127.21.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:46013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:06:41.663218 17995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:06:41.663563 17995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:06:41.663746 17995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:06:41.672178 17995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:06:41.672317 17995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:06:41.672449 17995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.3
I20260807 07:06:41.683318 17995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:46013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.3.keytab
--principal=kudu/127.21.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.83.3
--webserver_port=0
--tserver_master_addrs=127.21.83.62:34935
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.3
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:06:41.685488 17995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:06:41.687141 17995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 07:06:41.697176 17790 heartbeater.cc:503] Master 127.21.83.62:34935 was elected leader, sending a full tablet report...
W20260807 07:06:41.702072 18009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:41.702061 18012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:41.701989 18008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:06:41.703060 17995 server_base.cc:1034] running on GCE node
I20260807 07:06:41.703786 17995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:06:41.705499 17995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:06:41.706887 17995 hybrid_clock.cc:648] HybridClock initialized: now 1786086401706746 us; error 75 us; skew 500 ppm
I20260807 07:06:41.707295 17995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:06:41 dist-test-slave-ncp9 krb5kdc[17450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086401, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:06:41.712460 17995 init.cc:377] Logged in from keytab as kudu/127.21.83.3@KRBTEST.COM (short username kudu)
I20260807 07:06:41.714395 17995 webserver.cc:460] Webserver started at http://127.21.83.3:41877/ using document root <none> and password file <none>
I20260807 07:06:41.715139 17995 fs_manager.cc:359] Metadata directory not provided
I20260807 07:06:41.715272 17995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:06:41.715627 17995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:06:41.718822 17995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-2/data/instance:
uuid: "1e2ced9aa211458699125e0c72cc0a14"
format_stamp: "Formatted at 2026-08-07 07:06:41 on dist-test-slave-ncp9"
server_key: "cc27381f373d6a5c42d534b673f07801"
server_key_iv: "dbb4d29713dbb9ea94d404cbbd23e7c1"
server_key_version: "encryptionkey@0"
I20260807 07:06:41.719705 17995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-2/wal/instance:
uuid: "1e2ced9aa211458699125e0c72cc0a14"
format_stamp: "Formatted at 2026-08-07 07:06:41 on dist-test-slave-ncp9"
server_key: "cc27381f373d6a5c42d534b673f07801"
server_key_iv: "dbb4d29713dbb9ea94d404cbbd23e7c1"
server_key_version: "encryptionkey@0"
I20260807 07:06:41.727560 17995 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.008s
I20260807 07:06:41.731833 18028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:41.733351 17995 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260807 07:06:41.733520 17995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-2/wal
uuid: "1e2ced9aa211458699125e0c72cc0a14"
format_stamp: "Formatted at 2026-08-07 07:06:41 on dist-test-slave-ncp9"
server_key: "cc27381f373d6a5c42d534b673f07801"
server_key_iv: "dbb4d29713dbb9ea94d404cbbd23e7c1"
server_key_version: "encryptionkey@0"
I20260807 07:06:41.733755 17995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:06:41.755250 17995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:06:41.759588 17995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:06:41.759877 17995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:06:41.760951 17995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:06:41.762305 17995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:06:41.762400 17995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:41.762953 17995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:06:41.763042 17995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:41.798426 17995 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.3:41169
I20260807 07:06:41.798477 18189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.3:41169 every 8 connection(s)
I20260807 07:06:41.799770 17995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
I20260807 07:06:41.801930 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 17995
I20260807 07:06:41.802134 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-2/wal/instance
I20260807 07:06:41.802738 21836 external_mini_cluster.cc:1428] Setting key e60d12351d17407668ff1e9c59da522b
Aug 07 07:06:41 dist-test-slave-ncp9 krb5kdc[17450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086401, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.3@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:06:41.822108 18194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:41.802520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:57039 (local address 127.21.83.62:34935)
0807 07:06:41.803008 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:41.803012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:41.804280 (+ 1268us) server_negotiation.cc:407] Connection header received
0807 07:06:41.806682 (+ 2402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:41.806687 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:41.806778 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:41.806872 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:41.808556 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:41.809410 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:41.811023 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:41.811437 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:41.811553 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:41.815492 (+ 3939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:41.815499 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:41.815503 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:06:41.815563 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:41.819093 (+ 3530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:41.819916 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:41.819920 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:41.819922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:41.820059 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:41.820416 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:41.820419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:41.820421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:41.820712 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:41.820830 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:41.821619 (+ 789us) server_negotiation.cc:299] Negotiation successful
0807 07:06:41.821806 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":214,"threads_started":1}
I20260807 07:06:41.823047 18195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:41.803220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:34935 (local address 127.21.83.3:57039)
0807 07:06:41.804113 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:41.804169 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0807 07:06:41.806399 (+ 2230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:06:41.807147 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:06:41.807156 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:06:41.807745 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:06:41.808372 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:41.808390 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:41.809578 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:41.809582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:41.810862 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:41.810875 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:41.811694 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:41.811703 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:41.811927 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:41.812689 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:06:41.812725 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:06:41.815187 (+ 2462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:06:41.819258 (+ 4071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:06:41.819268 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:06:41.819284 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:06:41.819809 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:06:41.820143 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:06:41.820146 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:06:41.820149 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:06:41.820337 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:06:41.820807 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:06:41.820813 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:06:41.821232 (+ 419us) client_negotiation.cc:769] Sending connection context
0807 07:06:41.821474 (+ 242us) client_negotiation.cc:240] Negotiation successful
0807 07:06:41.821919 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":356,"threads_started":1}
I20260807 07:06:41.824599 18191 heartbeater.cc:347] Connected to a master server at 127.21.83.62:34935
I20260807 07:06:41.824926 18191 heartbeater.cc:464] Registering TS with master...
I20260807 07:06:41.825671 18191 heartbeater.cc:511] Master 127.21.83.62:34935 requested a full tablet report, sending...
I20260807 07:06:41.827169 17509 ts_manager.cc:194] Registered new tserver with Master: 1e2ced9aa211458699125e0c72cc0a14 (127.21.83.3:41169)
I20260807 07:06:41.827975 17509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.83.3@KRBTEST.COM'} at 127.21.83.3:57039
I20260807 07:06:41.831243 21836 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 07:06:41.848101 18194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:41.835494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45660 (local address 127.21.83.62:34935)
0807 07:06:41.835675 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:41.835679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:41.836126 (+ 447us) server_negotiation.cc:407] Connection header received
0807 07:06:41.836291 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:41.836295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:41.836360 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:41.836435 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:41.837993 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:41.838782 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:41.839985 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:41.840379 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:41.840466 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:41.842790 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:41.842795 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:41.842799 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:06:41.842833 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:41.845331 (+ 2498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:41.846042 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:41.846046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:41.846048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:41.846179 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:41.846654 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:41.846659 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:41.846661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:41.847009 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:41.847221 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:41.847661 (+ 440us) server_negotiation.cc:299] Negotiation successful
0807 07:06:41.847855 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 07:06:41.852653 17509 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45660:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 07:06:41.859730 17509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 07:06:41.901242 18218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:41.874647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41640 (local address 127.21.83.1:41367)
0807 07:06:41.891579 (+ 16932us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:41.891589 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:41.892973 (+ 1384us) server_negotiation.cc:407] Connection header received
0807 07:06:41.893096 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:41.893104 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:41.893339 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:41.893508 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:06:41.894348 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:41.895784 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:41.898956 (+ 3172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:41.899984 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:41.900120 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:41.900389 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:41.900695 (+ 306us) server_negotiation.cc:299] Negotiation successful
0807 07:06:41.900884 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16742,"thread_start_us":16584,"threads_started":1}
I20260807 07:06:41.901371 18214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:41.874034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:41367 (local address 127.0.0.1:41640)
0807 07:06:41.892708 (+ 18674us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:41.892726 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 07:06:41.892842 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:06:41.893829 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:06:41.893838 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:06:41.893861 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:06:41.894199 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:41.894208 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:41.895995 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:41.896003 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:41.898789 (+ 2786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:41.898801 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:41.900213 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:41.900220 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:41.900439 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:41.900488 (+ 49us) client_negotiation.cc:769] Sending connection context
0807 07:06:41.900613 (+ 125us) client_negotiation.cc:240] Negotiation successful
0807 07:06:41.900836 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18567,"mutex_wait_us":48,"thread_start_us":17743,"threads_started":1}
I20260807 07:06:41.902271 18213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:41.873683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.2:32993 (local address 127.0.0.1:54726)
0807 07:06:41.890799 (+ 17116us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:41.890857 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0807 07:06:41.892093 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:06:41.893200 (+ 1107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:06:41.893209 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:06:41.893234 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:06:41.893627 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:41.893637 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:41.895617 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:41.895622 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:41.898549 (+ 2927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:41.898566 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:41.900680 (+ 2114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:41.900687 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:41.900846 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:41.900880 (+ 34us) client_negotiation.cc:769] Sending connection context
0807 07:06:41.900993 (+ 113us) client_negotiation.cc:240] Negotiation successful
0807 07:06:41.901149 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16658,"mutex_wait_us":294,"thread_start_us":16457,"threads_started":1}
I20260807 07:06:41.902545 18212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:41.873715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54726 (local address 127.21.83.2:32993)
0807 07:06:41.886794 (+ 13079us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:41.886806 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:41.891044 (+ 4238us) server_negotiation.cc:407] Connection header received
0807 07:06:41.892400 (+ 1356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:41.892410 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:41.892706 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:41.892882 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:06:41.893796 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:41.895439 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:41.898736 (+ 3297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:41.900183 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:41.900306 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:41.900507 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:41.901915 (+ 1408us) server_negotiation.cc:299] Negotiation successful
0807 07:06:41.902126 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12727,"thread_start_us":12576,"threads_started":1}
I20260807 07:06:41.903508 18215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:41.874256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54412 (local address 127.21.83.3:41169)
0807 07:06:41.892349 (+ 18093us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:41.892357 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:41.892389 (+ 32us) server_negotiation.cc:407] Connection header received
0807 07:06:41.892481 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:41.892488 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:41.892734 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:41.892948 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:06:41.894074 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:41.895694 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:41.898526 (+ 2832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:41.900076 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:41.901767 (+ 1691us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:41.901997 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:41.902920 (+ 923us) server_negotiation.cc:299] Negotiation successful
0807 07:06:41.903139 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17900,"thread_start_us":18032,"threads_started":1}
I20260807 07:06:41.906289 17696 tablet_service.cc:1467] Processing CreateTablet for tablet 60f59cdf82ec4a7fb0612a205d784a1d (DEFAULT_TABLE table=test-table [id=6704af3e4b16490e873de9c5fdfd1f48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:06:41.906289 17891 tablet_service.cc:1467] Processing CreateTablet for tablet 60f59cdf82ec4a7fb0612a205d784a1d (DEFAULT_TABLE table=test-table [id=6704af3e4b16490e873de9c5fdfd1f48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:06:41.907886 17891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60f59cdf82ec4a7fb0612a205d784a1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:06:41.907886 17696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60f59cdf82ec4a7fb0612a205d784a1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:06:41.908623 18216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:41.873713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.3:41169 (local address 127.0.0.1:54412)
0807 07:06:41.891938 (+ 18225us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:41.891959 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 07:06:41.892116 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:06:41.893257 (+ 1141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:06:41.893261 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:06:41.893286 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:06:41.893854 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:41.893866 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:41.895879 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:41.895884 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:41.898339 (+ 2455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:41.898356 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:41.902223 (+ 3867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:41.902232 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:41.902649 (+ 417us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:41.902685 (+ 36us) client_negotiation.cc:769] Sending connection context
0807 07:06:41.902768 (+ 83us) client_negotiation.cc:240] Negotiation successful
0807 07:06:41.908304 (+ 5536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18070,"mutex_wait_us":206,"thread_start_us":17910,"threads_started":1}
I20260807 07:06:41.912467 18094 tablet_service.cc:1467] Processing CreateTablet for tablet 60f59cdf82ec4a7fb0612a205d784a1d (DEFAULT_TABLE table=test-table [id=6704af3e4b16490e873de9c5fdfd1f48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:06:41.913992 18094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60f59cdf82ec4a7fb0612a205d784a1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:06:41.926414 18225 tablet_bootstrap.cc:492] T 60f59cdf82ec4a7fb0612a205d784a1d P 1e2ced9aa211458699125e0c72cc0a14: Bootstrap starting.
I20260807 07:06:41.926458 18221 tablet_bootstrap.cc:492] T 60f59cdf82ec4a7fb0612a205d784a1d P f04d64dce74e4cf5a73f023cfa711d51: Bootstrap starting.
I20260807 07:06:41.925985 18219 tablet_bootstrap.cc:492] T 60f59cdf82ec4a7fb0612a205d784a1d P 1533cea80c9f4aef9cf7f9d06f90a0b4: Bootstrap starting.
I20260807 07:06:41.929863 18225 tablet_bootstrap.cc:654] T 60f59cdf82ec4a7fb0612a205d784a1d P 1e2ced9aa211458699125e0c72cc0a14: Neither blocks nor log segments found. Creating new log.
I20260807 07:06:41.930197 18221 tablet_bootstrap.cc:654] T 60f59cdf82ec4a7fb0612a205d784a1d P f04d64dce74e4cf5a73f023cfa711d51: Neither blocks nor log segments found. Creating new log.
I20260807 07:06:41.930323 18219 tablet_bootstrap.cc:654] T 60f59cdf82ec4a7fb0612a205d784a1d P 1533cea80c9f4aef9cf7f9d06f90a0b4: Neither blocks nor log segments found. Creating new log.
I20260807 07:06:41.930943 18225 log.cc:824] T 60f59cdf82ec4a7fb0612a205d784a1d P 1e2ced9aa211458699125e0c72cc0a14: Log is configured to *not* fsync() on all Append() calls
I20260807 07:06:41.931448 18219 log.cc:824] T 60f59cdf82ec4a7fb0612a205d784a1d P 1533cea80c9f4aef9cf7f9d06f90a0b4: Log is configured to *not* fsync() on all Append() calls
I20260807 07:06:41.931448 18221 log.cc:824] T 60f59cdf82ec4a7fb0612a205d784a1d P f04d64dce74e4cf5a73f023cfa711d51: Log is configured to *not* fsync() on all Append() calls
I20260807 07:06:41.933969 18225 tablet_bootstrap.cc:492] T 60f59cdf82ec4a7fb0612a205d784a1d P 1e2ced9aa211458699125e0c72cc0a14: No bootstrap required, opened a new log
I20260807 07:06:41.934221 18225 ts_tablet_manager.cc:1397] T 60f59cdf82ec4a7fb0612a205d784a1d P 1e2ced9aa211458699125e0c72cc0a14: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260807 07:06:41.934898 18221 tablet_bootstrap.cc:492] T 60f59cdf82ec4a7fb0612a205d784a1d P f04d64dce74e4cf5a73f023cfa711d51: No bootstrap required, opened a new log
I20260807 07:06:41.935149 18221 ts_tablet_manager.cc:1397] T 60f59cdf82ec4a7fb0612a205d784a1d P f04d64dce74e4cf5a73f023cfa711d51: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260807 07:06:41.939157 18219 tablet_bootstrap.cc:492] T 60f59cdf82ec4a7fb0612a205d784a1d P 1533cea80c9f4aef9cf7f9d06f90a0b4: No bootstrap required, opened a new log
I20260807 07:06:41.939440 18219 ts_tablet_manager.cc:1397] T 60f59cdf82ec4a7fb0612a205d784a1d P 1533cea80c9f4aef9cf7f9d06f90a0b4: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.006s
I20260807 07:06:41.941848 18225 raft_consensus.cc:348] T 60f59cdf82ec4a7fb0612a205d784a1d P 1e2ced9aa211458699125e0c72cc0a14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1533cea80c9f4aef9cf7f9d06f90a0b4" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 32993 } } peers { permanent_uuid: "1e2ced9aa211458699125e0c72cc0a14" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 41169 } } peers { permanent_uuid: "f04d64dce74e4cf5a73f023cfa711d51" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 41367 } }
I20260807 07:06:41.942224 18225 raft_consensus.cc:374] T 60f59cdf82ec4a7fb0612a205d784a1d P 1e2ced9aa211458699125e0c72cc0a14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:06:41.942358 18225 raft_consensus.cc:729] T 60f59cdf82ec4a7fb0612a205d784a1d P 1e2ced9aa211458699125e0c72cc0a14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2ced9aa211458699125e0c72cc0a14, State: Initialized, Role: FOLLOWER
I20260807 07:06:41.942940 18225 consensus_queue.cc:260] T 60f59cdf82ec4a7fb0612a205d784a1d P 1e2ced9aa211458699125e0c72cc0a14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1533cea80c9f4aef9cf7f9d06f90a0b4" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 32993 } } peers { permanent_uuid: "1e2ced9aa211458699125e0c72cc0a14" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 41169 } } peers { permanent_uuid: "f04d64dce74e4cf5a73f023cfa711d51" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 41367 } }
I20260807 07:06:41.943977 18191 heartbeater.cc:503] Master 127.21.83.62:34935 was elected leader, sending a full tablet report...
I20260807 07:06:41.945094 18221 raft_consensus.cc:348] T 60f59cdf82ec4a7fb0612a205d784a1d P f04d64dce74e4cf5a73f023cfa711d51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1533cea80c9f4aef9cf7f9d06f90a0b4" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 32993 } } peers { permanent_uuid: "1e2ced9aa211458699125e0c72cc0a14" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 41169 } } peers { permanent_uuid: "f04d64dce74e4cf5a73f023cfa711d51" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 41367 } }
I20260807 07:06:41.945645 18221 raft_consensus.cc:374] T 60f59cdf82ec4a7fb0612a205d784a1d P f04d64dce74e4cf5a73f023cfa711d51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:06:41.945786 18221 raft_consensus.cc:729] T 60f59cdf82ec4a7fb0612a205d784a1d P f04d64dce74e4cf5a73f023cfa711d51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f04d64dce74e4cf5a73f023cfa711d51, State: Initialized, Role: FOLLOWER
I20260807 07:06:41.945916 18225 ts_tablet_manager.cc:1428] T 60f59cdf82ec4a7fb0612a205d784a1d P 1e2ced9aa211458699125e0c72cc0a14: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20260807 07:06:41.946573 18221 consensus_queue.cc:260] T 60f59cdf82ec4a7fb0612a205d784a1d P f04d64dce74e4cf5a73f023cfa711d51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1533cea80c9f4aef9cf7f9d06f90a0b4" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 32993 } } peers { permanent_uuid: "1e2ced9aa211458699125e0c72cc0a14" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 41169 } } peers { permanent_uuid: "f04d64dce74e4cf5a73f023cfa711d51" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 41367 } }
I20260807 07:06:41.949215 18221 ts_tablet_manager.cc:1428] T 60f59cdf82ec4a7fb0612a205d784a1d P f04d64dce74e4cf5a73f023cfa711d51: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
I20260807 07:06:41.950960 18219 raft_consensus.cc:348] T 60f59cdf82ec4a7fb0612a205d784a1d P 1533cea80c9f4aef9cf7f9d06f90a0b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1533cea80c9f4aef9cf7f9d06f90a0b4" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 32993 } } peers { permanent_uuid: "1e2ced9aa211458699125e0c72cc0a14" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 41169 } } peers { permanent_uuid: "f04d64dce74e4cf5a73f023cfa711d51" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 41367 } }
I20260807 07:06:41.951419 18219 raft_consensus.cc:374] T 60f59cdf82ec4a7fb0612a205d784a1d P 1533cea80c9f4aef9cf7f9d06f90a0b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:06:41.951522 18219 raft_consensus.cc:729] T 60f59cdf82ec4a7fb0612a205d784a1d P 1533cea80c9f4aef9cf7f9d06f90a0b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1533cea80c9f4aef9cf7f9d06f90a0b4, State: Initialized, Role: FOLLOWER
I20260807 07:06:41.952081 18219 consensus_queue.cc:260] T 60f59cdf82ec4a7fb0612a205d784a1d P 1533cea80c9f4aef9cf7f9d06f90a0b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1533cea80c9f4aef9cf7f9d06f90a0b4" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 32993 } } peers { permanent_uuid: "1e2ced9aa211458699125e0c72cc0a14" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 41169 } } peers { permanent_uuid: "f04d64dce74e4cf5a73f023cfa711d51" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 41367 } }
I20260807 07:06:41.953106 17988 heartbeater.cc:503] Master 127.21.83.62:34935 was elected leader, sending a full tablet report...
I20260807 07:06:41.953611 18219 ts_tablet_manager.cc:1428] T 60f59cdf82ec4a7fb0612a205d784a1d P 1533cea80c9f4aef9cf7f9d06f90a0b4: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260807 07:06:41.968868 18233 raft_consensus.cc:482] T 60f59cdf82ec4a7fb0612a205d784a1d P f04d64dce74e4cf5a73f023cfa711d51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:06:41.969226 18233 raft_consensus.cc:504] T 60f59cdf82ec4a7fb0612a205d784a1d P f04d64dce74e4cf5a73f023cfa711d51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1533cea80c9f4aef9cf7f9d06f90a0b4" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 32993 } } peers { permanent_uuid: "1e2ced9aa211458699125e0c72cc0a14" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 41169 } } peers { permanent_uuid: "f04d64dce74e4cf5a73f023cfa711d51" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 41367 } }
I20260807 07:06:41.970974 18233 leader_election.cc:288] T 60f59cdf82ec4a7fb0612a205d784a1d P f04d64dce74e4cf5a73f023cfa711d51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1533cea80c9f4aef9cf7f9d06f90a0b4 (127.21.83.2:32993), 1e2ced9aa211458699125e0c72cc0a14 (127.21.83.3:41169)
W20260807 07:06:41.974912 17990 tablet.cc:2369] T 60f59cdf82ec4a7fb0612a205d784a1d P 1533cea80c9f4aef9cf7f9d06f90a0b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 07:06:41.978948 18212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:41.971527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:33091 (local address 127.21.83.2:32993)
0807 07:06:41.971991 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:41.971998 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:41.972251 (+ 253us) server_negotiation.cc:407] Connection header received
0807 07:06:41.972478 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:41.972482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:41.972542 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:41.972628 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:06:41.973288 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:41.974260 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:41.976819 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:41.978012 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:41.978143 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:41.978248 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:41.978552 (+ 304us) server_negotiation.cc:299] Negotiation successful
0807 07:06:41.978693 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302}
I20260807 07:06:41.979112 18236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:41.971267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.2:32993 (local address 127.21.83.1:33091)
0807 07:06:41.972133 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:41.972153 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 07:06:41.972326 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:06:41.972712 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:06:41.972716 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:06:41.972752 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:06:41.973144 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:41.973154 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:41.974444 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:41.974450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:41.976619 (+ 2169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:41.976637 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:41.978249 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:41.978256 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:41.978392 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:41.978417 (+ 25us) client_negotiation.cc:769] Sending connection context
0807 07:06:41.978512 (+ 95us) client_negotiation.cc:240] Negotiation successful
0807 07:06:41.978659 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"mutex_wait_us":51,"thread_start_us":519,"threads_started":1}
I20260807 07:06:41.979204 18215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:41.971567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:34769 (local address 127.21.83.3:41169)
0807 07:06:41.971783 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:41.971789 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:41.972626 (+ 837us) server_negotiation.cc:407] Connection header received
0807 07:06:41.972805 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:41.972810 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:41.972870 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:41.972981 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:06:41.973544 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:41.974371 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:41.977422 (+ 3051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:41.978279 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:41.978404 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:41.978509 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:41.978821 (+ 312us) server_negotiation.cc:299] Negotiation successful
0807 07:06:41.978957 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260807 07:06:41.979437 18237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:41.971510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.3:41169 (local address 127.21.83.1:34769)
0807 07:06:41.972513 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:41.972527 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 07:06:41.972654 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:06:41.973000 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:06:41.973010 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:06:41.973034 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:06:41.973393 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:41.973403 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:41.974669 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:41.974696 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:41.977180 (+ 2484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:41.977197 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:41.978517 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:41.978524 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:41.978667 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:41.978691 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 07:06:41.978774 (+ 83us) client_negotiation.cc:240] Negotiation successful
0807 07:06:41.978913 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":901,"mutex_wait_us":1,"thread_start_us":637,"threads_started":1}
I20260807 07:06:41.980402 17921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60f59cdf82ec4a7fb0612a205d784a1d" candidate_uuid: "f04d64dce74e4cf5a73f023cfa711d51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1533cea80c9f4aef9cf7f9d06f90a0b4" is_pre_election: true
I20260807 07:06:41.980444 18122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60f59cdf82ec4a7fb0612a205d784a1d" candidate_uuid: "f04d64dce74e4cf5a73f023cfa711d51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ced9aa211458699125e0c72cc0a14" is_pre_election: true
I20260807 07:06:41.980948 18122 raft_consensus.cc:2454] T 60f59cdf82ec4a7fb0612a205d784a1d P 1e2ced9aa211458699125e0c72cc0a14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f04d64dce74e4cf5a73f023cfa711d51 in term 0.
I20260807 07:06:41.980942 17921 raft_consensus.cc:2454] T 60f59cdf82ec4a7fb0612a205d784a1d P 1533cea80c9f4aef9cf7f9d06f90a0b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f04d64dce74e4cf5a73f023cfa711d51 in term 0.
I20260807 07:06:41.981626 17617 leader_election.cc:302] T 60f59cdf82ec4a7fb0612a205d784a1d P f04d64dce74e4cf5a73f023cfa711d51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1533cea80c9f4aef9cf7f9d06f90a0b4, f04d64dce74e4cf5a73f023cfa711d51; no voters:
I20260807 07:06:41.982024 18233 raft_consensus.cc:2781] T 60f59cdf82ec4a7fb0612a205d784a1d P f04d64dce74e4cf5a73f023cfa711d51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 07:06:41.982158 18233 raft_consensus.cc:482] T 60f59cdf82ec4a7fb0612a205d784a1d P f04d64dce74e4cf5a73f023cfa711d51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:06:41.982285 18233 raft_consensus.cc:3037] T 60f59cdf82ec4a7fb0612a205d784a1d P f04d64dce74e4cf5a73f023cfa711d51 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:06:41.984615 18233 raft_consensus.cc:504] T 60f59cdf82ec4a7fb0612a205d784a1d P f04d64dce74e4cf5a73f023cfa711d51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1533cea80c9f4aef9cf7f9d06f90a0b4" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 32993 } } peers { permanent_uuid: "1e2ced9aa211458699125e0c72cc0a14" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 41169 } } peers { permanent_uuid: "f04d64dce74e4cf5a73f023cfa711d51" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 41367 } }
I20260807 07:06:41.985332 18233 leader_election.cc:288] T 60f59cdf82ec4a7fb0612a205d784a1d P f04d64dce74e4cf5a73f023cfa711d51 [CANDIDATE]: Term 1 election: Requested vote from peers 1533cea80c9f4aef9cf7f9d06f90a0b4 (127.21.83.2:32993), 1e2ced9aa211458699125e0c72cc0a14 (127.21.83.3:41169)
I20260807 07:06:41.985949 18122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60f59cdf82ec4a7fb0612a205d784a1d" candidate_uuid: "f04d64dce74e4cf5a73f023cfa711d51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ced9aa211458699125e0c72cc0a14"
I20260807 07:06:41.985970 17921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60f59cdf82ec4a7fb0612a205d784a1d" candidate_uuid: "f04d64dce74e4cf5a73f023cfa711d51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1533cea80c9f4aef9cf7f9d06f90a0b4"
I20260807 07:06:41.986171 18122 raft_consensus.cc:3037] T 60f59cdf82ec4a7fb0612a205d784a1d P 1e2ced9aa211458699125e0c72cc0a14 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:06:41.986194 17921 raft_consensus.cc:3037] T 60f59cdf82ec4a7fb0612a205d784a1d P 1533cea80c9f4aef9cf7f9d06f90a0b4 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:06:41.988459 17921 raft_consensus.cc:2454] T 60f59cdf82ec4a7fb0612a205d784a1d P 1533cea80c9f4aef9cf7f9d06f90a0b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f04d64dce74e4cf5a73f023cfa711d51 in term 1.
I20260807 07:06:41.988462 18122 raft_consensus.cc:2454] T 60f59cdf82ec4a7fb0612a205d784a1d P 1e2ced9aa211458699125e0c72cc0a14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f04d64dce74e4cf5a73f023cfa711d51 in term 1.
I20260807 07:06:41.988981 17617 leader_election.cc:302] T 60f59cdf82ec4a7fb0612a205d784a1d P f04d64dce74e4cf5a73f023cfa711d51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1533cea80c9f4aef9cf7f9d06f90a0b4, 1e2ced9aa211458699125e0c72cc0a14, f04d64dce74e4cf5a73f023cfa711d51; no voters:
I20260807 07:06:41.989346 18233 raft_consensus.cc:2781] T 60f59cdf82ec4a7fb0612a205d784a1d P f04d64dce74e4cf5a73f023cfa711d51 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:06:41.989823 18233 raft_consensus.cc:686] T 60f59cdf82ec4a7fb0612a205d784a1d P f04d64dce74e4cf5a73f023cfa711d51 [term 1 LEADER]: Becoming Leader. State: Replica: f04d64dce74e4cf5a73f023cfa711d51, State: Running, Role: LEADER
I20260807 07:06:41.990296 18233 consensus_queue.cc:237] T 60f59cdf82ec4a7fb0612a205d784a1d P f04d64dce74e4cf5a73f023cfa711d51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1533cea80c9f4aef9cf7f9d06f90a0b4" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 32993 } } peers { permanent_uuid: "1e2ced9aa211458699125e0c72cc0a14" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 41169 } } peers { permanent_uuid: "f04d64dce74e4cf5a73f023cfa711d51" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 41367 } }
I20260807 07:06:41.994849 17509 catalog_manager.cc:5495] T 60f59cdf82ec4a7fb0612a205d784a1d P f04d64dce74e4cf5a73f023cfa711d51 reported cstate change: term changed from 0 to 1, leader changed from <none> to f04d64dce74e4cf5a73f023cfa711d51 (127.21.83.1). New cstate: current_term: 1 leader_uuid: "f04d64dce74e4cf5a73f023cfa711d51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1533cea80c9f4aef9cf7f9d06f90a0b4" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 32993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e2ced9aa211458699125e0c72cc0a14" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 41169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f04d64dce74e4cf5a73f023cfa711d51" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 41367 } health_report { overall_health: HEALTHY } } }
I20260807 07:06:42.033119 18218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:42.027911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41654 (local address 127.21.83.1:41367)
0807 07:06:42.028124 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:42.028128 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:42.028143 (+ 15us) server_negotiation.cc:407] Connection header received
0807 07:06:42.028228 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:42.028232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:42.028324 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:42.028452 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:06:42.028842 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:42.029616 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:42.030952 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:42.031342 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:42.031450 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:42.031982 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:06:42.032490 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:06:42.032634 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:42.032749 (+ 115us) server_negotiation.cc:299] Negotiation successful
0807 07:06:42.032833 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 07:06:42.041452 18122 raft_consensus.cc:1261] T 60f59cdf82ec4a7fb0612a205d784a1d P 1e2ced9aa211458699125e0c72cc0a14 [term 1 FOLLOWER]: Refusing update from remote peer f04d64dce74e4cf5a73f023cfa711d51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:06:42.041543 17921 raft_consensus.cc:1261] T 60f59cdf82ec4a7fb0612a205d784a1d P 1533cea80c9f4aef9cf7f9d06f90a0b4 [term 1 FOLLOWER]: Refusing update from remote peer f04d64dce74e4cf5a73f023cfa711d51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:06:42.042428 18238 consensus_queue.cc:1035] T 60f59cdf82ec4a7fb0612a205d784a1d P f04d64dce74e4cf5a73f023cfa711d51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2ced9aa211458699125e0c72cc0a14" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 41169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:06:42.042831 18233 consensus_queue.cc:1035] T 60f59cdf82ec4a7fb0612a205d784a1d P f04d64dce74e4cf5a73f023cfa711d51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1533cea80c9f4aef9cf7f9d06f90a0b4" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 07:06:42.051317 18192 tablet.cc:2369] T 60f59cdf82ec4a7fb0612a205d784a1d P 1e2ced9aa211458699125e0c72cc0a14: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 07:06:42.067997 17509 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:06:42.068241 17509 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:06:42.071619 17509 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 253f3a8ce6bd4ff3946e29a181278b9d: Sending DeleteTablet for 3 replicas of tablet 60f59cdf82ec4a7fb0612a205d784a1d
I20260807 07:06:42.072686 17891 tablet_service.cc:1514] Processing DeleteTablet for tablet 60f59cdf82ec4a7fb0612a205d784a1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:06:42 UTC) from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:54726
I20260807 07:06:42.072734 18094 tablet_service.cc:1514] Processing DeleteTablet for tablet 60f59cdf82ec4a7fb0612a205d784a1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:06:42 UTC) from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:54412
I20260807 07:06:42.073484 17696 tablet_service.cc:1514] Processing DeleteTablet for tablet 60f59cdf82ec4a7fb0612a205d784a1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:06:42 UTC) from {username='kudu', principal='kudu/127.21.83.62@KRBTEST.COM'} at 127.0.0.1:41640
I20260807 07:06:42.073578 18258 tablet_replica.cc:330] T 60f59cdf82ec4a7fb0612a205d784a1d P 1533cea80c9f4aef9cf7f9d06f90a0b4: stopping tablet replica
I20260807 07:06:42.074085 18258 raft_consensus.cc:2229] T 60f59cdf82ec4a7fb0612a205d784a1d P 1533cea80c9f4aef9cf7f9d06f90a0b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:06:42.074698 18258 raft_consensus.cc:2258] T 60f59cdf82ec4a7fb0612a205d784a1d P 1533cea80c9f4aef9cf7f9d06f90a0b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:06:42.074811 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 17599
I20260807 07:06:42.077235 18258 ts_tablet_manager.cc:1905] T 60f59cdf82ec4a7fb0612a205d784a1d P 1533cea80c9f4aef9cf7f9d06f90a0b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:06:42.081569 18261 tablet_replica.cc:330] T 60f59cdf82ec4a7fb0612a205d784a1d P f04d64dce74e4cf5a73f023cfa711d51: stopping tablet replica
I20260807 07:06:42.086637 18258 ts_tablet_manager.cc:1918] T 60f59cdf82ec4a7fb0612a205d784a1d P 1533cea80c9f4aef9cf7f9d06f90a0b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:06:42.087392 18261 raft_consensus.cc:2229] T 60f59cdf82ec4a7fb0612a205d784a1d P f04d64dce74e4cf5a73f023cfa711d51 [term 1 LEADER]: Raft consensus shutting down.
I20260807 07:06:42.088243 18261 raft_consensus.cc:2258] T 60f59cdf82ec4a7fb0612a205d784a1d P f04d64dce74e4cf5a73f023cfa711d51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:06:42.088850 18258 log.cc:1196] T 60f59cdf82ec4a7fb0612a205d784a1d P 1533cea80c9f4aef9cf7f9d06f90a0b4: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-1/wal/wals/60f59cdf82ec4a7fb0612a205d784a1d
I20260807 07:06:42.089574 18258 ts_tablet_manager.cc:1939] T 60f59cdf82ec4a7fb0612a205d784a1d P 1533cea80c9f4aef9cf7f9d06f90a0b4: Deleting consensus metadata
I20260807 07:06:42.090987 18261 ts_tablet_manager.cc:1905] T 60f59cdf82ec4a7fb0612a205d784a1d P f04d64dce74e4cf5a73f023cfa711d51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:06:42.091707 17488 catalog_manager.cc:4841] TS 1533cea80c9f4aef9cf7f9d06f90a0b4 (127.21.83.2:32993): tablet 60f59cdf82ec4a7fb0612a205d784a1d (table test-table [id=6704af3e4b16490e873de9c5fdfd1f48]) successfully deleted
I20260807 07:06:42.097405 18260 tablet_replica.cc:330] T 60f59cdf82ec4a7fb0612a205d784a1d P 1e2ced9aa211458699125e0c72cc0a14: stopping tablet replica
I20260807 07:06:42.097882 18260 raft_consensus.cc:2229] T 60f59cdf82ec4a7fb0612a205d784a1d P 1e2ced9aa211458699125e0c72cc0a14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:06:42.098417 18260 raft_consensus.cc:2258] T 60f59cdf82ec4a7fb0612a205d784a1d P 1e2ced9aa211458699125e0c72cc0a14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:06:42.098483 18261 ts_tablet_manager.cc:1918] T 60f59cdf82ec4a7fb0612a205d784a1d P f04d64dce74e4cf5a73f023cfa711d51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:06:42.098657 18261 log.cc:1196] T 60f59cdf82ec4a7fb0612a205d784a1d P f04d64dce74e4cf5a73f023cfa711d51: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-0/wal/wals/60f59cdf82ec4a7fb0612a205d784a1d
I20260807 07:06:42.099457 18261 ts_tablet_manager.cc:1939] T 60f59cdf82ec4a7fb0612a205d784a1d P f04d64dce74e4cf5a73f023cfa711d51: Deleting consensus metadata
I20260807 07:06:42.100682 18260 ts_tablet_manager.cc:1905] T 60f59cdf82ec4a7fb0612a205d784a1d P 1e2ced9aa211458699125e0c72cc0a14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:06:42.101384 17490 catalog_manager.cc:4841] TS f04d64dce74e4cf5a73f023cfa711d51 (127.21.83.1:41367): tablet 60f59cdf82ec4a7fb0612a205d784a1d (table test-table [id=6704af3e4b16490e873de9c5fdfd1f48]) successfully deleted
Aug 07 07:06:42 dist-test-slave-ncp9 krb5kdc[17450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.1@KRBTEST.COM
I20260807 07:06:42.107565 18260 ts_tablet_manager.cc:1918] T 60f59cdf82ec4a7fb0612a205d784a1d P 1e2ced9aa211458699125e0c72cc0a14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:06:42.107767 18260 log.cc:1196] T 60f59cdf82ec4a7fb0612a205d784a1d P 1e2ced9aa211458699125e0c72cc0a14: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786086078080656-21836-0/minicluster-data/ts-2/wal/wals/60f59cdf82ec4a7fb0612a205d784a1d
I20260807 07:06:42.108295 18260 ts_tablet_manager.cc:1939] T 60f59cdf82ec4a7fb0612a205d784a1d P 1e2ced9aa211458699125e0c72cc0a14: Deleting consensus metadata
I20260807 07:06:42.110201 17491 catalog_manager.cc:4841] TS 1e2ced9aa211458699125e0c72cc0a14 (127.21.83.3:41169): tablet 60f59cdf82ec4a7fb0612a205d784a1d (table test-table [id=6704af3e4b16490e873de9c5fdfd1f48]) successfully deleted
I20260807 07:06:42.110805 18218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:42.076264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41668 (local address 127.21.83.1:41367)
0807 07:06:42.076455 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:42.076465 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:42.086768 (+ 10303us) server_negotiation.cc:407] Connection header received
0807 07:06:42.087302 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:42.087314 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:42.087407 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:42.087488 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:42.093847 (+ 6359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:42.095044 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:42.099275 (+ 4231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:42.099936 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:42.100045 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:42.104632 (+ 4587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:42.104644 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:42.104656 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:06:42.104694 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:42.107534 (+ 2840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:42.108266 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:42.108273 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:42.108276 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:42.108432 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:42.108795 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:42.108799 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:42.108801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:42.109213 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:42.109498 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:42.109808 (+ 310us) server_negotiation.cc:299] Negotiation successful
0807 07:06:42.110552 (+ 744us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 07:06:42.111531 17782 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:06:42.212397 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 17599
I20260807 07:06:42.236480 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 17796
Aug 07 07:06:42 dist-test-slave-ncp9 krb5kdc[17450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.2@KRBTEST.COM
I20260807 07:06:42.251662 18212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:42.237446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54742 (local address 127.21.83.2:32993)
0807 07:06:42.237638 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:42.237644 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:42.237665 (+ 21us) server_negotiation.cc:407] Connection header received
0807 07:06:42.237722 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:42.237726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:42.237808 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:42.237899 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:42.239191 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:42.239932 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:42.241241 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:42.241645 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:42.241752 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:42.245958 (+ 4206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:42.245973 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:42.245990 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:06:42.246028 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:42.248832 (+ 2804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:42.249616 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:42.249624 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:42.249626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:42.249776 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:42.250150 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:42.250155 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:42.250158 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:42.250652 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:42.250896 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:42.251135 (+ 239us) server_negotiation.cc:299] Negotiation successful
0807 07:06:42.251313 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 07:06:42.252389 17981 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:06:42.342978 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 17796
I20260807 07:06:42.366173 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 17995
Aug 07 07:06:42 dist-test-slave-ncp9 krb5kdc[17450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.3@KRBTEST.COM
I20260807 07:06:42.381040 18215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:42.366918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54428 (local address 127.21.83.3:41169)
0807 07:06:42.367119 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:42.367125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:42.367146 (+ 21us) server_negotiation.cc:407] Connection header received
0807 07:06:42.367307 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:42.367313 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:42.367397 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:42.367502 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:42.368854 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:42.369645 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:42.370937 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:42.371338 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:42.371445 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:42.375900 (+ 4455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:42.375916 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:42.375931 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:06:42.375969 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:42.378604 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:42.379240 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:42.379248 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:42.379250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:42.379396 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:42.379687 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:42.379689 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:42.379691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:42.380056 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:42.380254 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:42.380508 (+ 254us) server_negotiation.cc:299] Negotiation successful
0807 07:06:42.380689 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 07:06:42.381682 18184 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:06:42.469453 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 17995
I20260807 07:06:42.491559 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 17468
I20260807 07:06:42.492650 17557 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:06:42.605983 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 17468
2026-08-07T07:06:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4636 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-07T07:06:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:06:42Z Disabled control of system clock
I20260807 07:06:42.706245 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:35957
--webserver_interface=127.21.83.62
--webserver_port=0
--builtin_ntp_servers=127.21.83.20:40115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:35957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 07:06:43.051071 18274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:06:43.051358 18274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:06:43.051447 18274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:06:43.058522 18274 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:06:43.058612 18274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:06:43.058655 18274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:06:43.058718 18274 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:06:43.058760 18274 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:06:43.067589 18274 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:40115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:35957
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:35957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:06:43.068778 18274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:06:43.070533 18274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:06:43.081815 18282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:43.081856 18284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:43.081771 18281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:06:43.082849 18274 server_base.cc:1034] running on GCE node
I20260807 07:06:43.083580 18274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:06:43.084986 18274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:06:43.086187 18274 hybrid_clock.cc:648] HybridClock initialized: now 1786086403086144 us; error 39 us; skew 500 ppm
I20260807 07:06:43.086510 18274 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 07:06:43.092032 18274 webserver.cc:460] Webserver started at http://127.21.83.62:34983/ using document root <none> and password file <none>
I20260807 07:06:43.092784 18274 fs_manager.cc:359] Metadata directory not provided
I20260807 07:06:43.092927 18274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:06:43.093253 18274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:06:43.096231 18274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "6fa25ae8ddbe41d88168a97dc4dae9c8"
format_stamp: "Formatted at 2026-08-07 07:06:43 on dist-test-slave-ncp9"
server_key: "56427617944518be3d88f652ff084e72"
server_key_iv: "b3240057d26017697de2325b91f03ec5"
server_key_version: "encryptionkey@0"
I20260807 07:06:43.097075 18274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "6fa25ae8ddbe41d88168a97dc4dae9c8"
format_stamp: "Formatted at 2026-08-07 07:06:43 on dist-test-slave-ncp9"
server_key: "56427617944518be3d88f652ff084e72"
server_key_iv: "b3240057d26017697de2325b91f03ec5"
server_key_version: "encryptionkey@0"
I20260807 07:06:43.102116 18274 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260807 07:06:43.105443 18293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:43.106642 18274 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 07:06:43.106802 18274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "6fa25ae8ddbe41d88168a97dc4dae9c8"
format_stamp: "Formatted at 2026-08-07 07:06:43 on dist-test-slave-ncp9"
server_key: "56427617944518be3d88f652ff084e72"
server_key_iv: "b3240057d26017697de2325b91f03ec5"
server_key_version: "encryptionkey@0"
I20260807 07:06:43.106989 18274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:06:43.125646 18274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:06:43.126459 18274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:06:43.126729 18274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:06:43.145766 18274 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.62:35957
I20260807 07:06:43.145773 18370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.62:35957 every 8 connection(s)
I20260807 07:06:43.147238 18274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:06:43.148507 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 18274
I20260807 07:06:43.148691 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:06:43.149284 21836 external_mini_cluster.cc:1428] Setting key 7c685c3dbe6f329417a2dc78d5226458
I20260807 07:06:43.152252 18371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:06:43.163558 18376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:43.151313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55528 (local address 127.21.83.62:35957)
0807 07:06:43.152417 (+ 1104us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:43.152429 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:43.152476 (+ 47us) server_negotiation.cc:407] Connection header received
0807 07:06:43.154231 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:43.154257 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:43.154745 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:43.155180 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:43.155865 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:43.157343 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:43.158783 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:43.159490 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:43.159662 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:43.161003 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:43.161012 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:43.161030 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:06:43.161675 (+ 645us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:43.161776 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:06:43.161801 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:43.161928 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:43.162176 (+ 248us) server_negotiation.cc:299] Negotiation successful
0807 07:06:43.162533 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":378,"threads_started":1}
I20260807 07:06:43.164099 18371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fa25ae8ddbe41d88168a97dc4dae9c8: Bootstrap starting.
I20260807 07:06:43.168802 18371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fa25ae8ddbe41d88168a97dc4dae9c8: Neither blocks nor log segments found. Creating new log.
I20260807 07:06:43.169947 18371 log.cc:824] T 00000000000000000000000000000000 P 6fa25ae8ddbe41d88168a97dc4dae9c8: Log is configured to *not* fsync() on all Append() calls
I20260807 07:06:43.173318 18371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fa25ae8ddbe41d88168a97dc4dae9c8: No bootstrap required, opened a new log
I20260807 07:06:43.181448 18371 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fa25ae8ddbe41d88168a97dc4dae9c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fa25ae8ddbe41d88168a97dc4dae9c8" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 35957 } }
I20260807 07:06:43.181869 18371 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fa25ae8ddbe41d88168a97dc4dae9c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:06:43.181969 18371 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fa25ae8ddbe41d88168a97dc4dae9c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fa25ae8ddbe41d88168a97dc4dae9c8, State: Initialized, Role: FOLLOWER
I20260807 07:06:43.182646 18371 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fa25ae8ddbe41d88168a97dc4dae9c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fa25ae8ddbe41d88168a97dc4dae9c8" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 35957 } }
I20260807 07:06:43.182881 18371 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fa25ae8ddbe41d88168a97dc4dae9c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:06:43.183023 18371 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fa25ae8ddbe41d88168a97dc4dae9c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:06:43.183187 18371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fa25ae8ddbe41d88168a97dc4dae9c8 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:06:43.185577 18371 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fa25ae8ddbe41d88168a97dc4dae9c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fa25ae8ddbe41d88168a97dc4dae9c8" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 35957 } }
I20260807 07:06:43.186158 18371 leader_election.cc:302] T 00000000000000000000000000000000 P 6fa25ae8ddbe41d88168a97dc4dae9c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fa25ae8ddbe41d88168a97dc4dae9c8; no voters:
I20260807 07:06:43.186848 18371 leader_election.cc:288] T 00000000000000000000000000000000 P 6fa25ae8ddbe41d88168a97dc4dae9c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 07:06:43.186976 18380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fa25ae8ddbe41d88168a97dc4dae9c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:06:43.187714 18380 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fa25ae8ddbe41d88168a97dc4dae9c8 [term 1 LEADER]: Becoming Leader. State: Replica: 6fa25ae8ddbe41d88168a97dc4dae9c8, State: Running, Role: LEADER
I20260807 07:06:43.188293 18380 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fa25ae8ddbe41d88168a97dc4dae9c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fa25ae8ddbe41d88168a97dc4dae9c8" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 35957 } }
I20260807 07:06:43.188989 18371 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fa25ae8ddbe41d88168a97dc4dae9c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:06:43.191022 18381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fa25ae8ddbe41d88168a97dc4dae9c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6fa25ae8ddbe41d88168a97dc4dae9c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fa25ae8ddbe41d88168a97dc4dae9c8" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 35957 } } }
I20260807 07:06:43.191458 18381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fa25ae8ddbe41d88168a97dc4dae9c8 [sys.catalog]: This master's current role is: LEADER
I20260807 07:06:43.191188 18382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fa25ae8ddbe41d88168a97dc4dae9c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fa25ae8ddbe41d88168a97dc4dae9c8. Latest consensus state: current_term: 1 leader_uuid: "6fa25ae8ddbe41d88168a97dc4dae9c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fa25ae8ddbe41d88168a97dc4dae9c8" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 35957 } } }
I20260807 07:06:43.191854 18382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fa25ae8ddbe41d88168a97dc4dae9c8 [sys.catalog]: This master's current role is: LEADER
I20260807 07:06:43.195274 18393 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:06:43.200596 18393 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:06:43.210186 18393 catalog_manager.cc:1347] Generated new cluster ID: 641139afa9674fcea777afd79e52cd03
I20260807 07:06:43.210304 18393 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:06:43.227169 18393 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:06:43.228600 18393 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:06:43.239080 18393 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6fa25ae8ddbe41d88168a97dc4dae9c8: Generated new TSK 0
I20260807 07:06:43.240077 18393 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 07:06:43.254325 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.1:0
--local_ip_for_outbound_sockets=127.21.83.1
--webserver_interface=127.21.83.1
--webserver_port=0
--tserver_master_addrs=127.21.83.62:35957
--builtin_ntp_servers=127.21.83.20:40115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 07:06:43.627063 18410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:06:43.627343 18410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:06:43.627429 18410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:06:43.634519 18410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:06:43.634618 18410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:06:43.634727 18410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.1
I20260807 07:06:43.643692 18410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:40115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.83.1
--webserver_port=0
--tserver_master_addrs=127.21.83.62:35957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.1
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:06:43.644933 18410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:06:43.646639 18410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:06:43.659431 18416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:43.659392 18418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:43.659478 18415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:06:43.665973 18410 server_base.cc:1034] running on GCE node
I20260807 07:06:43.666973 18410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:06:43.668179 18410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:06:43.669418 18410 hybrid_clock.cc:648] HybridClock initialized: now 1786086403669362 us; error 49 us; skew 500 ppm
I20260807 07:06:43.669763 18410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 07:06:43.672271 18410 webserver.cc:460] Webserver started at http://127.21.83.1:44935/ using document root <none> and password file <none>
I20260807 07:06:43.673017 18410 fs_manager.cc:359] Metadata directory not provided
I20260807 07:06:43.673128 18410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:06:43.673435 18410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:06:43.676334 18410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-0/data/instance:
uuid: "f2027115fef940f3bab570cdffe6a947"
format_stamp: "Formatted at 2026-08-07 07:06:43 on dist-test-slave-ncp9"
server_key: "c21b04173fc6e4aaaea45937bcea114a"
server_key_iv: "0b9e863d363a8dab3cf4efbc6510b255"
server_key_version: "encryptionkey@0"
I20260807 07:06:43.677177 18410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-0/wal/instance:
uuid: "f2027115fef940f3bab570cdffe6a947"
format_stamp: "Formatted at 2026-08-07 07:06:43 on dist-test-slave-ncp9"
server_key: "c21b04173fc6e4aaaea45937bcea114a"
server_key_iv: "0b9e863d363a8dab3cf4efbc6510b255"
server_key_version: "encryptionkey@0"
I20260807 07:06:43.682098 18410 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 07:06:43.685366 18426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:43.686570 18410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 07:06:43.686718 18410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-0/wal
uuid: "f2027115fef940f3bab570cdffe6a947"
format_stamp: "Formatted at 2026-08-07 07:06:43 on dist-test-slave-ncp9"
server_key: "c21b04173fc6e4aaaea45937bcea114a"
server_key_iv: "0b9e863d363a8dab3cf4efbc6510b255"
server_key_version: "encryptionkey@0"
I20260807 07:06:43.686906 18410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:06:43.702710 18410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:06:43.703624 18410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:06:43.703907 18410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:06:43.705180 18410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:06:43.706982 18410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:06:43.707091 18410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:43.707212 18410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:06:43.707283 18410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:43.738950 18410 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.1:39981
I20260807 07:06:43.739012 18598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.1:39981 every 8 connection(s)
I20260807 07:06:43.740216 18410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-0/data/info.pb
I20260807 07:06:43.745707 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 18410
I20260807 07:06:43.745863 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-0/wal/instance
I20260807 07:06:43.746407 21836 external_mini_cluster.cc:1428] Setting key e8312e3d15ecce80848e731d96c03b60
I20260807 07:06:43.749506 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.2:0
--local_ip_for_outbound_sockets=127.21.83.2
--webserver_interface=127.21.83.2
--webserver_port=0
--tserver_master_addrs=127.21.83.62:35957
--builtin_ntp_servers=127.21.83.20:40115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 07:06:43.753366 18605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:43.742590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.1:57063 (local address 127.21.83.62:35957)
0807 07:06:43.742987 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:43.742992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:43.744018 (+ 1026us) server_negotiation.cc:407] Connection header received
0807 07:06:43.745831 (+ 1813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:43.745836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:43.745915 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:43.746016 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:43.747327 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:43.748332 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:43.750513 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:43.750952 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:43.751051 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:43.752599 (+ 1548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:43.752603 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:43.752607 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:06:43.752626 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:43.752685 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:06:43.752697 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:43.752772 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:43.753011 (+ 239us) server_negotiation.cc:299] Negotiation successful
0807 07:06:43.753094 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":207,"threads_started":1}
I20260807 07:06:43.754699 18606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:43.743041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:35957 (local address 127.21.83.1:57063)
0807 07:06:43.743867 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:43.743909 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 07:06:43.745581 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:06:43.746298 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:06:43.746309 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:06:43.746416 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:06:43.747155 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:43.747180 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:43.748530 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:43.748535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:43.750361 (+ 1826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:43.750376 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:43.751258 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:43.751266 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:43.751544 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:43.752261 (+ 717us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 07:06:43.752264 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:06:43.752314 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 07:06:43.752411 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:06:43.752783 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:06:43.752794 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:06:43.752801 (+ 7us) client_negotiation.cc:769] Sending connection context
0807 07:06:43.753005 (+ 204us) client_negotiation.cc:240] Negotiation successful
0807 07:06:43.753389 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":256,"threads_started":1}
I20260807 07:06:43.756282 18600 heartbeater.cc:347] Connected to a master server at 127.21.83.62:35957
I20260807 07:06:43.756621 18600 heartbeater.cc:464] Registering TS with master...
I20260807 07:06:43.757637 18600 heartbeater.cc:511] Master 127.21.83.62:35957 requested a full tablet report, sending...
I20260807 07:06:43.759919 18317 ts_manager.cc:194] Registered new tserver with Master: f2027115fef940f3bab570cdffe6a947 (127.21.83.1:39981)
I20260807 07:06:43.761533 18317 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.83.1:57063
W20260807 07:06:44.113935 18609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:06:44.114266 18609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:06:44.114405 18609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:06:44.122085 18609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:06:44.122246 18609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:06:44.122403 18609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.2
I20260807 07:06:44.132546 18609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:40115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.83.2
--webserver_port=0
--tserver_master_addrs=127.21.83.62:35957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.2
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:06:44.133908 18609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:06:44.135690 18609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:06:44.149386 18619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:44.149421 18616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:44.153529 18617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:06:44.154690 18609 server_base.cc:1034] running on GCE node
I20260807 07:06:44.155495 18609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:06:44.156762 18609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:06:44.158022 18609 hybrid_clock.cc:648] HybridClock initialized: now 1786086404157933 us; error 104 us; skew 500 ppm
I20260807 07:06:44.158406 18609 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 07:06:44.161360 18609 webserver.cc:460] Webserver started at http://127.21.83.2:33651/ using document root <none> and password file <none>
I20260807 07:06:44.162262 18609 fs_manager.cc:359] Metadata directory not provided
I20260807 07:06:44.162397 18609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:06:44.162746 18609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:06:44.167321 18609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-1/data/instance:
uuid: "7a549b13aef34432b6fcf27bbc208b69"
format_stamp: "Formatted at 2026-08-07 07:06:44 on dist-test-slave-ncp9"
server_key: "5f2cb66892f095dbada7daac3c31ea05"
server_key_iv: "ae114760f27eac7ace735af1e28d43ce"
server_key_version: "encryptionkey@0"
I20260807 07:06:44.168780 18609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-1/wal/instance:
uuid: "7a549b13aef34432b6fcf27bbc208b69"
format_stamp: "Formatted at 2026-08-07 07:06:44 on dist-test-slave-ncp9"
server_key: "5f2cb66892f095dbada7daac3c31ea05"
server_key_iv: "ae114760f27eac7ace735af1e28d43ce"
server_key_version: "encryptionkey@0"
I20260807 07:06:44.174662 18609 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 07:06:44.178689 18625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:44.180269 18609 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 07:06:44.180459 18609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-1/wal
uuid: "7a549b13aef34432b6fcf27bbc208b69"
format_stamp: "Formatted at 2026-08-07 07:06:44 on dist-test-slave-ncp9"
server_key: "5f2cb66892f095dbada7daac3c31ea05"
server_key_iv: "ae114760f27eac7ace735af1e28d43ce"
server_key_version: "encryptionkey@0"
I20260807 07:06:44.180704 18609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:06:44.194017 18609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:06:44.194904 18609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:06:44.195210 18609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:06:44.196346 18609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:06:44.198148 18609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:06:44.198254 18609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:44.198374 18609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:06:44.198447 18609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:44.231546 18609 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.2:34337
I20260807 07:06:44.231601 18791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.2:34337 every 8 connection(s)
I20260807 07:06:44.232857 18609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-1/data/info.pb
I20260807 07:06:44.242769 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 18609
I20260807 07:06:44.242966 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-1/wal/instance
I20260807 07:06:44.243395 21836 external_mini_cluster.cc:1428] Setting key 75069c42b8dabff1878df086161bc02f
I20260807 07:06:44.246201 18605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:44.235278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.2:55119 (local address 127.21.83.62:35957)
0807 07:06:44.235449 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:44.235453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:44.236811 (+ 1358us) server_negotiation.cc:407] Connection header received
0807 07:06:44.238645 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:44.238650 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:44.238742 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:44.238833 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:44.239967 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:44.240719 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:44.242432 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:44.243073 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:44.243236 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:44.245249 (+ 2013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:44.245254 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:44.245258 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:06:44.245278 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:44.245314 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:06:44.245327 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:44.245413 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:44.245750 (+ 337us) server_negotiation.cc:299] Negotiation successful
0807 07:06:44.245864 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 07:06:44.247233 18796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:44.235792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:35957 (local address 127.21.83.2:55119)
0807 07:06:44.236653 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:44.236697 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 07:06:44.238365 (+ 1668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:06:44.239080 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:06:44.239089 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:06:44.239186 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:06:44.239792 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:44.239811 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:44.240875 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:44.240879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:44.242266 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:44.242282 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:44.243354 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:44.243364 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:44.243918 (+ 554us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:44.244814 (+ 896us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 07:06:44.244818 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:06:44.244882 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 07:06:44.245033 (+ 151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:06:44.245436 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:06:44.245454 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:06:44.245462 (+ 8us) client_negotiation.cc:769] Sending connection context
0807 07:06:44.245677 (+ 215us) client_negotiation.cc:240] Negotiation successful
0807 07:06:44.246072 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":274,"threads_started":1}
I20260807 07:06:44.247139 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.83.3:0
--local_ip_for_outbound_sockets=127.21.83.3
--webserver_interface=127.21.83.3
--webserver_port=0
--tserver_master_addrs=127.21.83.62:35957
--builtin_ntp_servers=127.21.83.20:40115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 07:06:44.249111 18793 heartbeater.cc:347] Connected to a master server at 127.21.83.62:35957
I20260807 07:06:44.249490 18793 heartbeater.cc:464] Registering TS with master...
I20260807 07:06:44.250460 18793 heartbeater.cc:511] Master 127.21.83.62:35957 requested a full tablet report, sending...
I20260807 07:06:44.252307 18317 ts_manager.cc:194] Registered new tserver with Master: 7a549b13aef34432b6fcf27bbc208b69 (127.21.83.2:34337)
I20260807 07:06:44.253334 18317 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.83.2:55119
W20260807 07:06:44.603775 18798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:06:44.604111 18798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:06:44.604254 18798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:06:44.612756 18798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:06:44.612927 18798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:06:44.613106 18798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.83.3
I20260807 07:06:44.622722 18798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:40115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.83.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.83.3
--webserver_port=0
--tserver_master_addrs=127.21.83.62:35957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.83.3
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:06:44.624054 18798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:06:44.625977 18798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:06:44.638824 18804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:44.638906 18805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:44.638960 18807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:06:44.640192 18798 server_base.cc:1034] running on GCE node
I20260807 07:06:44.640802 18798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:06:44.641733 18798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:06:44.642938 18798 hybrid_clock.cc:648] HybridClock initialized: now 1786086404642882 us; error 48 us; skew 500 ppm
I20260807 07:06:44.643230 18798 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 07:06:44.645435 18798 webserver.cc:460] Webserver started at http://127.21.83.3:40989/ using document root <none> and password file <none>
I20260807 07:06:44.646075 18798 fs_manager.cc:359] Metadata directory not provided
I20260807 07:06:44.646173 18798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:06:44.646471 18798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:06:44.649240 18798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-2/data/instance:
uuid: "8366bea66c834ba8a93e7c6619ed6050"
format_stamp: "Formatted at 2026-08-07 07:06:44 on dist-test-slave-ncp9"
server_key: "fa1c1aafa06da5285cfe044c7a3ed5a3"
server_key_iv: "b9d417eeb19fc750bbd23a99d37cc5a8"
server_key_version: "encryptionkey@0"
I20260807 07:06:44.649998 18798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-2/wal/instance:
uuid: "8366bea66c834ba8a93e7c6619ed6050"
format_stamp: "Formatted at 2026-08-07 07:06:44 on dist-test-slave-ncp9"
server_key: "fa1c1aafa06da5285cfe044c7a3ed5a3"
server_key_iv: "b9d417eeb19fc750bbd23a99d37cc5a8"
server_key_version: "encryptionkey@0"
I20260807 07:06:44.654876 18798 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 07:06:44.658200 18813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:44.659466 18798 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 07:06:44.659622 18798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-2/wal
uuid: "8366bea66c834ba8a93e7c6619ed6050"
format_stamp: "Formatted at 2026-08-07 07:06:44 on dist-test-slave-ncp9"
server_key: "fa1c1aafa06da5285cfe044c7a3ed5a3"
server_key_iv: "b9d417eeb19fc750bbd23a99d37cc5a8"
server_key_version: "encryptionkey@0"
I20260807 07:06:44.659832 18798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:06:44.688012 18798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:06:44.688862 18798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:06:44.689162 18798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:06:44.690249 18798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:06:44.691835 18798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:06:44.691933 18798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:44.692047 18798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:06:44.692117 18798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:44.723644 18798 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.3:38249
I20260807 07:06:44.723716 18988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.3:38249 every 8 connection(s)
I20260807 07:06:44.724967 18798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-2/data/info.pb
I20260807 07:06:44.726316 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 18798
I20260807 07:06:44.726516 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-2/wal/instance
I20260807 07:06:44.727099 21836 external_mini_cluster.cc:1428] Setting key d03630858a478f0276d42e665014ff89
I20260807 07:06:44.737937 18605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:44.727740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:41747 (local address 127.21.83.62:35957)
0807 07:06:44.727928 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:44.727934 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:44.729687 (+ 1753us) server_negotiation.cc:407] Connection header received
0807 07:06:44.731672 (+ 1985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:44.731677 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:44.731764 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:44.731847 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:44.732956 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:44.733701 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:44.735304 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:44.735715 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:44.735823 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:44.737139 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:44.737144 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:44.737148 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:06:44.737165 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:44.737197 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:06:44.737208 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:44.737281 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:44.737497 (+ 216us) server_negotiation.cc:299] Negotiation successful
0807 07:06:44.737603 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 07:06:44.739105 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:44.728415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:35957 (local address 127.21.83.3:41747)
0807 07:06:44.729544 (+ 1129us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:44.729588 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 07:06:44.731395 (+ 1807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:06:44.732102 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:06:44.732110 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:06:44.732199 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:06:44.732773 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:44.732794 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:44.733853 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:44.733857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:44.735163 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:44.735176 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:44.735933 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:44.735941 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:44.736161 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:44.736806 (+ 645us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 07:06:44.736809 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:06:44.736850 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 07:06:44.736962 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:06:44.737294 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:06:44.737307 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:06:44.737315 (+ 8us) client_negotiation.cc:769] Sending connection context
0807 07:06:44.737473 (+ 158us) client_negotiation.cc:240] Negotiation successful
0807 07:06:44.737843 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":372,"threads_started":1}
I20260807 07:06:44.740367 18989 heartbeater.cc:347] Connected to a master server at 127.21.83.62:35957
I20260807 07:06:44.740660 18989 heartbeater.cc:464] Registering TS with master...
I20260807 07:06:44.741555 18989 heartbeater.cc:511] Master 127.21.83.62:35957 requested a full tablet report, sending...
I20260807 07:06:44.742975 18317 ts_manager.cc:194] Registered new tserver with Master: 8366bea66c834ba8a93e7c6619ed6050 (127.21.83.3:38249)
I20260807 07:06:44.743860 18317 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.83.3:41747
I20260807 07:06:44.754570 21836 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 07:06:44.764286 18605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:44.758716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55530 (local address 127.21.83.62:35957)
0807 07:06:44.758881 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:44.758886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:44.759312 (+ 426us) server_negotiation.cc:407] Connection header received
0807 07:06:44.759446 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:44.759450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:44.759511 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:44.759589 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:44.760095 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:44.760766 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:44.762023 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:44.762437 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:44.762526 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:44.763586 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:44.763593 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:44.763597 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:06:44.763615 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:44.763666 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:06:44.763683 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:44.763776 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:44.763942 (+ 166us) server_negotiation.cc:299] Negotiation successful
0807 07:06:44.764022 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260807 07:06:44.766274 18600 heartbeater.cc:503] Master 127.21.83.62:35957 was elected leader, sending a full tablet report...
I20260807 07:06:44.768121 18316 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55530:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 07:06:44.774832 18316 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 07:06:44.809063 19007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:44.786251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51004 (local address 127.21.83.2:34337)
0807 07:06:44.799889 (+ 13638us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:44.799898 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:44.799941 (+ 43us) server_negotiation.cc:407] Connection header received
0807 07:06:44.800072 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:44.800081 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:44.800382 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:44.801088 (+ 706us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:06:44.801603 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:44.802955 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:44.806195 (+ 3240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:44.807319 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:44.808141 (+ 822us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:44.808385 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:44.808489 (+ 104us) server_negotiation.cc:299] Negotiation successful
0807 07:06:44.808673 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13490,"thread_start_us":13390,"threads_started":1}
I20260807 07:06:44.809151 19006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:44.788521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.3:38249 (local address 127.0.0.1:48998)
0807 07:06:44.800387 (+ 11866us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:44.800406 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 07:06:44.800552 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:06:44.801526 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:06:44.801530 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:06:44.801556 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:06:44.802011 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:44.802022 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:44.803541 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:44.803551 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:44.805921 (+ 2370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:44.805940 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:44.808242 (+ 2302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:44.808249 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:44.808434 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:44.808467 (+ 33us) client_negotiation.cc:769] Sending connection context
0807 07:06:44.808560 (+ 93us) client_negotiation.cc:240] Negotiation successful
0807 07:06:44.808710 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11736,"mutex_wait_us":71,"thread_start_us":10672,"threads_started":1}
I20260807 07:06:44.809379 19005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:44.786019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.2:34337 (local address 127.0.0.1:51004)
0807 07:06:44.786799 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:44.786836 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0807 07:06:44.787014 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:06:44.800697 (+ 13683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:06:44.800705 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:06:44.800750 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:06:44.801336 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:44.801356 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:44.803205 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:44.803210 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:44.805921 (+ 2711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:44.805940 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:44.807546 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:44.807552 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:44.807759 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:44.807823 (+ 64us) client_negotiation.cc:769] Sending connection context
0807 07:06:44.807946 (+ 123us) client_negotiation.cc:240] Negotiation successful
0807 07:06:44.808597 (+ 651us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"mutex_wait_us":60,"thread_start_us":14060,"threads_started":1}
I20260807 07:06:44.810009 19009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:44.786139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:39981 (local address 127.0.0.1:35652)
0807 07:06:44.799639 (+ 13500us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:44.799661 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 07:06:44.799855 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:06:44.800883 (+ 1028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:06:44.800887 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:06:44.800934 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:06:44.801957 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:44.801974 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:44.804069 (+ 2095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:44.804078 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:44.806310 (+ 2232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:44.806327 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:44.808551 (+ 2224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:44.808557 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:44.808718 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:44.808748 (+ 30us) client_negotiation.cc:769] Sending connection context
0807 07:06:44.808814 (+ 66us) client_negotiation.cc:240] Negotiation successful
0807 07:06:44.808956 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13203,"mutex_wait_us":25,"thread_start_us":626,"threads_started":1}
I20260807 07:06:44.810161 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:44.789262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48998 (local address 127.21.83.3:38249)
0807 07:06:44.798977 (+ 9715us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:44.798986 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:44.800706 (+ 1720us) server_negotiation.cc:407] Connection header received
0807 07:06:44.800818 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:44.800825 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:44.801088 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:44.801286 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:06:44.802247 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:44.803360 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:44.806231 (+ 2871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:44.807618 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:44.807756 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:44.808082 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:44.809607 (+ 1525us) server_negotiation.cc:299] Negotiation successful
0807 07:06:44.809788 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9529,"thread_start_us":9356,"threads_started":1}
I20260807 07:06:44.810451 19008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:44.786629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35652 (local address 127.21.83.1:39981)
0807 07:06:44.798177 (+ 11548us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:44.798189 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:44.800004 (+ 1815us) server_negotiation.cc:407] Connection header received
0807 07:06:44.800103 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:44.800120 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:44.800397 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:44.800606 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:06:44.802218 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:44.803896 (+ 1678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:44.806537 (+ 2641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:44.807998 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:44.808138 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:44.808409 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:44.809612 (+ 1203us) server_negotiation.cc:299] Negotiation successful
0807 07:06:44.809788 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11268,"thread_start_us":11584,"threads_started":1}
I20260807 07:06:44.813773 18887 tablet_service.cc:1467] Processing CreateTablet for tablet 7f302f05a39b4b7f8999fdee31f1e69c (DEFAULT_TABLE table=test-table [id=99cb842e7f674a9487bdffa6ac822a97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:06:44.813786 18493 tablet_service.cc:1467] Processing CreateTablet for tablet 7f302f05a39b4b7f8999fdee31f1e69c (DEFAULT_TABLE table=test-table [id=99cb842e7f674a9487bdffa6ac822a97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:06:44.813786 18693 tablet_service.cc:1467] Processing CreateTablet for tablet 7f302f05a39b4b7f8999fdee31f1e69c (DEFAULT_TABLE table=test-table [id=99cb842e7f674a9487bdffa6ac822a97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 07:06:44.815088 18887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f302f05a39b4b7f8999fdee31f1e69c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:06:44.815115 18493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f302f05a39b4b7f8999fdee31f1e69c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:06:44.815137 18693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f302f05a39b4b7f8999fdee31f1e69c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:06:44.826655 19011 tablet_bootstrap.cc:492] T 7f302f05a39b4b7f8999fdee31f1e69c P 8366bea66c834ba8a93e7c6619ed6050: Bootstrap starting.
I20260807 07:06:44.829648 19012 tablet_bootstrap.cc:492] T 7f302f05a39b4b7f8999fdee31f1e69c P f2027115fef940f3bab570cdffe6a947: Bootstrap starting.
I20260807 07:06:44.829901 19011 tablet_bootstrap.cc:654] T 7f302f05a39b4b7f8999fdee31f1e69c P 8366bea66c834ba8a93e7c6619ed6050: Neither blocks nor log segments found. Creating new log.
I20260807 07:06:44.831086 19017 tablet_bootstrap.cc:492] T 7f302f05a39b4b7f8999fdee31f1e69c P 7a549b13aef34432b6fcf27bbc208b69: Bootstrap starting.
I20260807 07:06:44.832222 19011 log.cc:824] T 7f302f05a39b4b7f8999fdee31f1e69c P 8366bea66c834ba8a93e7c6619ed6050: Log is configured to *not* fsync() on all Append() calls
I20260807 07:06:44.834151 19017 tablet_bootstrap.cc:654] T 7f302f05a39b4b7f8999fdee31f1e69c P 7a549b13aef34432b6fcf27bbc208b69: Neither blocks nor log segments found. Creating new log.
I20260807 07:06:44.834271 19012 tablet_bootstrap.cc:654] T 7f302f05a39b4b7f8999fdee31f1e69c P f2027115fef940f3bab570cdffe6a947: Neither blocks nor log segments found. Creating new log.
I20260807 07:06:44.835099 19017 log.cc:824] T 7f302f05a39b4b7f8999fdee31f1e69c P 7a549b13aef34432b6fcf27bbc208b69: Log is configured to *not* fsync() on all Append() calls
I20260807 07:06:44.835443 19012 log.cc:824] T 7f302f05a39b4b7f8999fdee31f1e69c P f2027115fef940f3bab570cdffe6a947: Log is configured to *not* fsync() on all Append() calls
I20260807 07:06:44.835587 19011 tablet_bootstrap.cc:492] T 7f302f05a39b4b7f8999fdee31f1e69c P 8366bea66c834ba8a93e7c6619ed6050: No bootstrap required, opened a new log
I20260807 07:06:44.835860 19011 ts_tablet_manager.cc:1397] T 7f302f05a39b4b7f8999fdee31f1e69c P 8366bea66c834ba8a93e7c6619ed6050: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260807 07:06:44.837862 19012 tablet_bootstrap.cc:492] T 7f302f05a39b4b7f8999fdee31f1e69c P f2027115fef940f3bab570cdffe6a947: No bootstrap required, opened a new log
I20260807 07:06:44.838027 19017 tablet_bootstrap.cc:492] T 7f302f05a39b4b7f8999fdee31f1e69c P 7a549b13aef34432b6fcf27bbc208b69: No bootstrap required, opened a new log
I20260807 07:06:44.838107 19012 ts_tablet_manager.cc:1397] T 7f302f05a39b4b7f8999fdee31f1e69c P f2027115fef940f3bab570cdffe6a947: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260807 07:06:44.838289 19017 ts_tablet_manager.cc:1397] T 7f302f05a39b4b7f8999fdee31f1e69c P 7a549b13aef34432b6fcf27bbc208b69: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260807 07:06:44.845182 19012 raft_consensus.cc:348] T 7f302f05a39b4b7f8999fdee31f1e69c P f2027115fef940f3bab570cdffe6a947 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a549b13aef34432b6fcf27bbc208b69" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 34337 } } peers { permanent_uuid: "f2027115fef940f3bab570cdffe6a947" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 39981 } } peers { permanent_uuid: "8366bea66c834ba8a93e7c6619ed6050" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 38249 } }
I20260807 07:06:44.845535 19012 raft_consensus.cc:374] T 7f302f05a39b4b7f8999fdee31f1e69c P f2027115fef940f3bab570cdffe6a947 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:06:44.845625 19012 raft_consensus.cc:729] T 7f302f05a39b4b7f8999fdee31f1e69c P f2027115fef940f3bab570cdffe6a947 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2027115fef940f3bab570cdffe6a947, State: Initialized, Role: FOLLOWER
I20260807 07:06:44.846185 19012 consensus_queue.cc:260] T 7f302f05a39b4b7f8999fdee31f1e69c P f2027115fef940f3bab570cdffe6a947 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a549b13aef34432b6fcf27bbc208b69" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 34337 } } peers { permanent_uuid: "f2027115fef940f3bab570cdffe6a947" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 39981 } } peers { permanent_uuid: "8366bea66c834ba8a93e7c6619ed6050" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 38249 } }
I20260807 07:06:44.846056 19011 raft_consensus.cc:348] T 7f302f05a39b4b7f8999fdee31f1e69c P 8366bea66c834ba8a93e7c6619ed6050 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a549b13aef34432b6fcf27bbc208b69" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 34337 } } peers { permanent_uuid: "f2027115fef940f3bab570cdffe6a947" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 39981 } } peers { permanent_uuid: "8366bea66c834ba8a93e7c6619ed6050" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 38249 } }
I20260807 07:06:44.846455 19011 raft_consensus.cc:374] T 7f302f05a39b4b7f8999fdee31f1e69c P 8366bea66c834ba8a93e7c6619ed6050 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:06:44.846557 19011 raft_consensus.cc:729] T 7f302f05a39b4b7f8999fdee31f1e69c P 8366bea66c834ba8a93e7c6619ed6050 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8366bea66c834ba8a93e7c6619ed6050, State: Initialized, Role: FOLLOWER
I20260807 07:06:44.847127 19011 consensus_queue.cc:260] T 7f302f05a39b4b7f8999fdee31f1e69c P 8366bea66c834ba8a93e7c6619ed6050 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a549b13aef34432b6fcf27bbc208b69" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 34337 } } peers { permanent_uuid: "f2027115fef940f3bab570cdffe6a947" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 39981 } } peers { permanent_uuid: "8366bea66c834ba8a93e7c6619ed6050" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 38249 } }
I20260807 07:06:44.847901 19012 ts_tablet_manager.cc:1428] T 7f302f05a39b4b7f8999fdee31f1e69c P f2027115fef940f3bab570cdffe6a947: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260807 07:06:44.848503 19017 raft_consensus.cc:348] T 7f302f05a39b4b7f8999fdee31f1e69c P 7a549b13aef34432b6fcf27bbc208b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a549b13aef34432b6fcf27bbc208b69" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 34337 } } peers { permanent_uuid: "f2027115fef940f3bab570cdffe6a947" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 39981 } } peers { permanent_uuid: "8366bea66c834ba8a93e7c6619ed6050" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 38249 } }
I20260807 07:06:44.848963 19017 raft_consensus.cc:374] T 7f302f05a39b4b7f8999fdee31f1e69c P 7a549b13aef34432b6fcf27bbc208b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:06:44.849069 19017 raft_consensus.cc:729] T 7f302f05a39b4b7f8999fdee31f1e69c P 7a549b13aef34432b6fcf27bbc208b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a549b13aef34432b6fcf27bbc208b69, State: Initialized, Role: FOLLOWER
I20260807 07:06:44.849160 18989 heartbeater.cc:503] Master 127.21.83.62:35957 was elected leader, sending a full tablet report...
I20260807 07:06:44.849785 19011 ts_tablet_manager.cc:1428] T 7f302f05a39b4b7f8999fdee31f1e69c P 8366bea66c834ba8a93e7c6619ed6050: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260807 07:06:44.849761 19017 consensus_queue.cc:260] T 7f302f05a39b4b7f8999fdee31f1e69c P 7a549b13aef34432b6fcf27bbc208b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a549b13aef34432b6fcf27bbc208b69" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 34337 } } peers { permanent_uuid: "f2027115fef940f3bab570cdffe6a947" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 39981 } } peers { permanent_uuid: "8366bea66c834ba8a93e7c6619ed6050" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 38249 } }
I20260807 07:06:44.851152 18793 heartbeater.cc:503] Master 127.21.83.62:35957 was elected leader, sending a full tablet report...
I20260807 07:06:44.851732 19017 ts_tablet_manager.cc:1428] T 7f302f05a39b4b7f8999fdee31f1e69c P 7a549b13aef34432b6fcf27bbc208b69: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260807 07:06:44.976672 18990 tablet.cc:2369] T 7f302f05a39b4b7f8999fdee31f1e69c P 8366bea66c834ba8a93e7c6619ed6050: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 07:06:44.985018 18795 tablet.cc:2369] T 7f302f05a39b4b7f8999fdee31f1e69c P 7a549b13aef34432b6fcf27bbc208b69: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 07:06:44.992676 18603 tablet.cc:2369] T 7f302f05a39b4b7f8999fdee31f1e69c P f2027115fef940f3bab570cdffe6a947: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 07:06:45.065043 19024 raft_consensus.cc:482] T 7f302f05a39b4b7f8999fdee31f1e69c P 8366bea66c834ba8a93e7c6619ed6050 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:06:45.065359 19024 raft_consensus.cc:504] T 7f302f05a39b4b7f8999fdee31f1e69c P 8366bea66c834ba8a93e7c6619ed6050 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a549b13aef34432b6fcf27bbc208b69" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 34337 } } peers { permanent_uuid: "f2027115fef940f3bab570cdffe6a947" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 39981 } } peers { permanent_uuid: "8366bea66c834ba8a93e7c6619ed6050" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 38249 } }
I20260807 07:06:45.067134 19024 leader_election.cc:288] T 7f302f05a39b4b7f8999fdee31f1e69c P 8366bea66c834ba8a93e7c6619ed6050 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a549b13aef34432b6fcf27bbc208b69 (127.21.83.2:34337), f2027115fef940f3bab570cdffe6a947 (127.21.83.1:39981)
I20260807 07:06:45.073553 19007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:45.067702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:50049 (local address 127.21.83.2:34337)
0807 07:06:45.067885 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:45.067891 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:45.067912 (+ 21us) server_negotiation.cc:407] Connection header received
0807 07:06:45.068161 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:45.068167 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:45.068259 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:45.068350 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:06:45.068806 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:45.069768 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:45.071695 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:45.072691 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:45.072786 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:45.072856 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:45.073234 (+ 378us) server_negotiation.cc:299] Negotiation successful
0807 07:06:45.073335 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 07:06:45.073617 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:45.067452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.2:34337 (local address 127.21.83.3:50049)
0807 07:06:45.067798 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:45.067818 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 07:06:45.067966 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:06:45.068425 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:06:45.068429 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:06:45.068458 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:06:45.068637 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:45.068649 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:45.069899 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:45.069905 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:45.071505 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:45.071518 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:45.072839 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:45.072845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:45.073042 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:45.073079 (+ 37us) client_negotiation.cc:769] Sending connection context
0807 07:06:45.073183 (+ 104us) client_negotiation.cc:240] Negotiation successful
0807 07:06:45.073326 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218}
I20260807 07:06:45.074669 19026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:45.067713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:39981 (local address 127.21.83.3:49767)
0807 07:06:45.068532 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:45.068546 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 07:06:45.068646 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:06:45.068958 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:06:45.068963 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:06:45.068988 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 07:06:45.069367 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:45.069379 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:45.070601 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:45.070606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:45.072195 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:06:45.072210 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:06:45.074078 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:06:45.074088 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:06:45.074256 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:45.074281 (+ 25us) client_negotiation.cc:769] Sending connection context
0807 07:06:45.074357 (+ 76us) client_negotiation.cc:240] Negotiation successful
0807 07:06:45.074443 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":421,"threads_started":1}
I20260807 07:06:45.074783 19008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:45.067845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.83.3:49767 (local address 127.21.83.1:39981)
0807 07:06:45.068049 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:45.068054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:45.068641 (+ 587us) server_negotiation.cc:407] Connection header received
0807 07:06:45.068742 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:45.068747 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:45.068836 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:45.068943 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 07:06:45.069528 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:45.070413 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:45.072402 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:45.073476 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:45.073596 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:45.073712 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:45.074424 (+ 712us) server_negotiation.cc:299] Negotiation successful
0807 07:06:45.074556 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 07:06:45.074764 18731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f302f05a39b4b7f8999fdee31f1e69c" candidate_uuid: "8366bea66c834ba8a93e7c6619ed6050" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a549b13aef34432b6fcf27bbc208b69" is_pre_election: true
I20260807 07:06:45.075380 18731 raft_consensus.cc:2454] T 7f302f05a39b4b7f8999fdee31f1e69c P 7a549b13aef34432b6fcf27bbc208b69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8366bea66c834ba8a93e7c6619ed6050 in term 0.
I20260807 07:06:45.075664 18528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f302f05a39b4b7f8999fdee31f1e69c" candidate_uuid: "8366bea66c834ba8a93e7c6619ed6050" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2027115fef940f3bab570cdffe6a947" is_pre_election: true
I20260807 07:06:45.076066 18820 leader_election.cc:302] T 7f302f05a39b4b7f8999fdee31f1e69c P 8366bea66c834ba8a93e7c6619ed6050 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a549b13aef34432b6fcf27bbc208b69, 8366bea66c834ba8a93e7c6619ed6050; no voters:
I20260807 07:06:45.076155 18528 raft_consensus.cc:2454] T 7f302f05a39b4b7f8999fdee31f1e69c P f2027115fef940f3bab570cdffe6a947 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8366bea66c834ba8a93e7c6619ed6050 in term 0.
I20260807 07:06:45.076445 19024 raft_consensus.cc:2781] T 7f302f05a39b4b7f8999fdee31f1e69c P 8366bea66c834ba8a93e7c6619ed6050 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 07:06:45.076577 19024 raft_consensus.cc:482] T 7f302f05a39b4b7f8999fdee31f1e69c P 8366bea66c834ba8a93e7c6619ed6050 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:06:45.076687 19024 raft_consensus.cc:3037] T 7f302f05a39b4b7f8999fdee31f1e69c P 8366bea66c834ba8a93e7c6619ed6050 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:06:45.078896 19024 raft_consensus.cc:504] T 7f302f05a39b4b7f8999fdee31f1e69c P 8366bea66c834ba8a93e7c6619ed6050 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a549b13aef34432b6fcf27bbc208b69" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 34337 } } peers { permanent_uuid: "f2027115fef940f3bab570cdffe6a947" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 39981 } } peers { permanent_uuid: "8366bea66c834ba8a93e7c6619ed6050" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 38249 } }
I20260807 07:06:45.079538 19024 leader_election.cc:288] T 7f302f05a39b4b7f8999fdee31f1e69c P 8366bea66c834ba8a93e7c6619ed6050 [CANDIDATE]: Term 1 election: Requested vote from peers 7a549b13aef34432b6fcf27bbc208b69 (127.21.83.2:34337), f2027115fef940f3bab570cdffe6a947 (127.21.83.1:39981)
I20260807 07:06:45.079893 18731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f302f05a39b4b7f8999fdee31f1e69c" candidate_uuid: "8366bea66c834ba8a93e7c6619ed6050" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a549b13aef34432b6fcf27bbc208b69"
I20260807 07:06:45.080050 18528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f302f05a39b4b7f8999fdee31f1e69c" candidate_uuid: "8366bea66c834ba8a93e7c6619ed6050" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2027115fef940f3bab570cdffe6a947"
I20260807 07:06:45.080111 18731 raft_consensus.cc:3037] T 7f302f05a39b4b7f8999fdee31f1e69c P 7a549b13aef34432b6fcf27bbc208b69 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:06:45.080237 18528 raft_consensus.cc:3037] T 7f302f05a39b4b7f8999fdee31f1e69c P f2027115fef940f3bab570cdffe6a947 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:06:45.082356 18731 raft_consensus.cc:2454] T 7f302f05a39b4b7f8999fdee31f1e69c P 7a549b13aef34432b6fcf27bbc208b69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8366bea66c834ba8a93e7c6619ed6050 in term 1.
I20260807 07:06:45.082521 18528 raft_consensus.cc:2454] T 7f302f05a39b4b7f8999fdee31f1e69c P f2027115fef940f3bab570cdffe6a947 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8366bea66c834ba8a93e7c6619ed6050 in term 1.
I20260807 07:06:45.082701 18820 leader_election.cc:302] T 7f302f05a39b4b7f8999fdee31f1e69c P 8366bea66c834ba8a93e7c6619ed6050 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a549b13aef34432b6fcf27bbc208b69, 8366bea66c834ba8a93e7c6619ed6050; no voters:
I20260807 07:06:45.083050 19024 raft_consensus.cc:2781] T 7f302f05a39b4b7f8999fdee31f1e69c P 8366bea66c834ba8a93e7c6619ed6050 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:06:45.083467 19024 raft_consensus.cc:686] T 7f302f05a39b4b7f8999fdee31f1e69c P 8366bea66c834ba8a93e7c6619ed6050 [term 1 LEADER]: Becoming Leader. State: Replica: 8366bea66c834ba8a93e7c6619ed6050, State: Running, Role: LEADER
I20260807 07:06:45.083904 19024 consensus_queue.cc:237] T 7f302f05a39b4b7f8999fdee31f1e69c P 8366bea66c834ba8a93e7c6619ed6050 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a549b13aef34432b6fcf27bbc208b69" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 34337 } } peers { permanent_uuid: "f2027115fef940f3bab570cdffe6a947" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 39981 } } peers { permanent_uuid: "8366bea66c834ba8a93e7c6619ed6050" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 38249 } }
I20260807 07:06:45.088490 18316 catalog_manager.cc:5495] T 7f302f05a39b4b7f8999fdee31f1e69c P 8366bea66c834ba8a93e7c6619ed6050 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8366bea66c834ba8a93e7c6619ed6050 (127.21.83.3). New cstate: current_term: 1 leader_uuid: "8366bea66c834ba8a93e7c6619ed6050" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a549b13aef34432b6fcf27bbc208b69" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 34337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2027115fef940f3bab570cdffe6a947" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 39981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8366bea66c834ba8a93e7c6619ed6050" member_type: VOTER last_known_addr { host: "127.21.83.3" port: 38249 } health_report { overall_health: HEALTHY } } }
I20260807 07:06:45.111704 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:45.103268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49002 (local address 127.21.83.3:38249)
0807 07:06:45.103438 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:45.103443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:45.103461 (+ 18us) server_negotiation.cc:407] Connection header received
0807 07:06:45.103624 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:45.103627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:45.103721 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:45.103852 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:06:45.104176 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:45.104976 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:45.106833 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:45.109436 (+ 2603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:45.109587 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:45.110314 (+ 727us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:06:45.111061 (+ 747us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:06:45.111221 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:45.111327 (+ 106us) server_negotiation.cc:299] Negotiation successful
0807 07:06:45.111428 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260807 07:06:45.121755 18528 raft_consensus.cc:1261] T 7f302f05a39b4b7f8999fdee31f1e69c P f2027115fef940f3bab570cdffe6a947 [term 1 FOLLOWER]: Refusing update from remote peer 8366bea66c834ba8a93e7c6619ed6050: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:06:45.122916 18731 raft_consensus.cc:1261] T 7f302f05a39b4b7f8999fdee31f1e69c P 7a549b13aef34432b6fcf27bbc208b69 [term 1 FOLLOWER]: Refusing update from remote peer 8366bea66c834ba8a93e7c6619ed6050: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:06:45.123057 19028 consensus_queue.cc:1035] T 7f302f05a39b4b7f8999fdee31f1e69c P 8366bea66c834ba8a93e7c6619ed6050 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2027115fef940f3bab570cdffe6a947" member_type: VOTER last_known_addr { host: "127.21.83.1" port: 39981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:06:45.123735 19024 consensus_queue.cc:1035] T 7f302f05a39b4b7f8999fdee31f1e69c P 8366bea66c834ba8a93e7c6619ed6050 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a549b13aef34432b6fcf27bbc208b69" member_type: VOTER last_known_addr { host: "127.21.83.2" port: 34337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:06:45.136783 19037 mvcc.cc:205] Tried to move back new op lower bound from 7315809915365928960 to 7315809915227303936. Current Snapshot: MvccSnapshot[applied={T|T < 7315809915365928960}]
I20260807 07:06:45.151391 18317 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:06:45.151705 18317 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 07:06:45.155964 18317 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6fa25ae8ddbe41d88168a97dc4dae9c8: Sending DeleteTablet for 3 replicas of tablet 7f302f05a39b4b7f8999fdee31f1e69c
I20260807 07:06:45.157155 18493 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f302f05a39b4b7f8999fdee31f1e69c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:06:45 UTC) from {username='slave'} at 127.0.0.1:35652
I20260807 07:06:45.157194 18693 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f302f05a39b4b7f8999fdee31f1e69c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:06:45 UTC) from {username='slave'} at 127.0.0.1:51004
I20260807 07:06:45.160251 18887 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f302f05a39b4b7f8999fdee31f1e69c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 07:06:45 UTC) from {username='slave'} at 127.0.0.1:48998
I20260807 07:06:45.162120 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 18410
I20260807 07:06:45.171896 19046 tablet_replica.cc:330] T 7f302f05a39b4b7f8999fdee31f1e69c P f2027115fef940f3bab570cdffe6a947: stopping tablet replica
I20260807 07:06:45.172396 19046 raft_consensus.cc:2229] T 7f302f05a39b4b7f8999fdee31f1e69c P f2027115fef940f3bab570cdffe6a947 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:06:45.173427 19008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:45.163679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35660 (local address 127.21.83.1:39981)
0807 07:06:45.163854 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:45.163860 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:45.163877 (+ 17us) server_negotiation.cc:407] Connection header received
0807 07:06:45.163934 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:45.163938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:45.164014 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:45.164110 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:45.164777 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:45.165954 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:45.169109 (+ 3155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:45.169887 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:45.170008 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:45.171633 (+ 1625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:45.171645 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:45.171669 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:06:45.172457 (+ 788us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:45.172527 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:06:45.172554 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:45.172655 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:45.172935 (+ 280us) server_negotiation.cc:299] Negotiation successful
0807 07:06:45.173077 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 07:06:45.173511 19045 tablet_replica.cc:330] T 7f302f05a39b4b7f8999fdee31f1e69c P 7a549b13aef34432b6fcf27bbc208b69: stopping tablet replica
I20260807 07:06:45.174233 19045 raft_consensus.cc:2229] T 7f302f05a39b4b7f8999fdee31f1e69c P 7a549b13aef34432b6fcf27bbc208b69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 07:06:45.174626 18593 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:06:45.174769 19045 raft_consensus.cc:2258] T 7f302f05a39b4b7f8999fdee31f1e69c P 7a549b13aef34432b6fcf27bbc208b69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:06:45.177227 19048 tablet_replica.cc:330] T 7f302f05a39b4b7f8999fdee31f1e69c P 8366bea66c834ba8a93e7c6619ed6050: stopping tablet replica
I20260807 07:06:45.177400 19045 ts_tablet_manager.cc:1905] T 7f302f05a39b4b7f8999fdee31f1e69c P 7a549b13aef34432b6fcf27bbc208b69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:06:45.177811 19048 raft_consensus.cc:2229] T 7f302f05a39b4b7f8999fdee31f1e69c P 8366bea66c834ba8a93e7c6619ed6050 [term 1 LEADER]: Raft consensus shutting down.
I20260807 07:06:45.178642 19048 raft_consensus.cc:2258] T 7f302f05a39b4b7f8999fdee31f1e69c P 8366bea66c834ba8a93e7c6619ed6050 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:06:45.180872 19048 ts_tablet_manager.cc:1905] T 7f302f05a39b4b7f8999fdee31f1e69c P 8366bea66c834ba8a93e7c6619ed6050: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 07:06:45.185614 19045 ts_tablet_manager.cc:1918] T 7f302f05a39b4b7f8999fdee31f1e69c P 7a549b13aef34432b6fcf27bbc208b69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:06:45.185793 19045 log.cc:1196] T 7f302f05a39b4b7f8999fdee31f1e69c P 7a549b13aef34432b6fcf27bbc208b69: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-1/wal/wals/7f302f05a39b4b7f8999fdee31f1e69c
I20260807 07:06:45.186352 19045 ts_tablet_manager.cc:1939] T 7f302f05a39b4b7f8999fdee31f1e69c P 7a549b13aef34432b6fcf27bbc208b69: Deleting consensus metadata
I20260807 07:06:45.188436 18297 catalog_manager.cc:4841] TS 7a549b13aef34432b6fcf27bbc208b69 (127.21.83.2:34337): tablet 7f302f05a39b4b7f8999fdee31f1e69c (table test-table [id=99cb842e7f674a9487bdffa6ac822a97]) successfully deleted
I20260807 07:06:45.189337 19048 ts_tablet_manager.cc:1918] T 7f302f05a39b4b7f8999fdee31f1e69c P 8366bea66c834ba8a93e7c6619ed6050: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 07:06:45.189561 19048 log.cc:1196] T 7f302f05a39b4b7f8999fdee31f1e69c P 8366bea66c834ba8a93e7c6619ed6050: Deleting WAL directory at /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786086078080656-21836-0/minicluster-data/ts-2/wal/wals/7f302f05a39b4b7f8999fdee31f1e69c
I20260807 07:06:45.190658 19048 ts_tablet_manager.cc:1939] T 7f302f05a39b4b7f8999fdee31f1e69c P 8366bea66c834ba8a93e7c6619ed6050: Deleting consensus metadata
I20260807 07:06:45.192451 18294 catalog_manager.cc:4841] TS 8366bea66c834ba8a93e7c6619ed6050 (127.21.83.3:38249): tablet 7f302f05a39b4b7f8999fdee31f1e69c (table test-table [id=99cb842e7f674a9487bdffa6ac822a97]) successfully deleted
I20260807 07:06:45.338717 19046 raft_consensus.cc:2258] T 7f302f05a39b4b7f8999fdee31f1e69c P f2027115fef940f3bab570cdffe6a947 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:06:45.339512 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 18410
W20260807 07:06:45.362673 18296 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.83.1:39981 (error 108)
I20260807 07:06:45.362811 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 18609
I20260807 07:06:45.364960 19052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:45.364151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:39981 (local address 127.0.0.1:35670)
0807 07:06:45.364604 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:45.364727 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.1:39981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":211,"threads_started":1}
W20260807 07:06:45.365180 18296 catalog_manager.cc:4584] TS f2027115fef940f3bab570cdffe6a947 (127.21.83.1:39981): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7f302f05a39b4b7f8999fdee31f1e69c: Network error: Client connection negotiation failed: client connection to 127.21.83.1:39981: connect: Connection refused (error 111)
I20260807 07:06:45.371425 19007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:45.363854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51006 (local address 127.21.83.2:34337)
0807 07:06:45.364052 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:45.364059 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:45.364082 (+ 23us) server_negotiation.cc:407] Connection header received
0807 07:06:45.364239 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:45.364245 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:45.364365 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:45.364491 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:45.365191 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:45.366001 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:45.368173 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:45.368621 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:45.368733 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:45.369955 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:45.369964 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:45.369983 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:06:45.370585 (+ 602us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:45.370635 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:06:45.370657 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:45.370770 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:45.370910 (+ 140us) server_negotiation.cc:299] Negotiation successful
0807 07:06:45.371055 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260807 07:06:45.372351 18787 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:06:45.419114 19052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:45.418534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:39981 (local address 127.0.0.1:35682)
0807 07:06:45.418809 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:45.418936 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.1:39981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260807 07:06:45.420418 19052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:45.419946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:39981 (local address 127.0.0.1:35686)
0807 07:06:45.420131 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:45.420227 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.1:39981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260807 07:06:45.476686 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 18609
I20260807 07:06:45.481092 19052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:45.480554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:39981 (local address 127.0.0.1:35688)
0807 07:06:45.480748 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:45.480856 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.1:39981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260807 07:06:45.482494 19052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:45.482091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:39981 (local address 127.0.0.1:35700)
0807 07:06:45.482252 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:45.482339 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.1:39981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260807 07:06:45.505820 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 18798
I20260807 07:06:45.514286 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:45.506611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49004 (local address 127.21.83.3:38249)
0807 07:06:45.506815 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:45.506822 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:45.507068 (+ 246us) server_negotiation.cc:407] Connection header received
0807 07:06:45.507292 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:45.507296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:45.507385 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:45.507476 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:45.507975 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:45.508753 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:45.510786 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:45.511303 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:45.511406 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:45.512636 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:45.512649 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:45.512664 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:06:45.513448 (+ 784us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:45.513501 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:06:45.513525 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:45.513612 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:45.513802 (+ 190us) server_negotiation.cc:299] Negotiation successful
0807 07:06:45.513906 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260807 07:06:45.515138 18982 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:06:45.563241 19052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:45.562753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:39981 (local address 127.0.0.1:35704)
0807 07:06:45.562989 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:45.563077 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.1:39981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260807 07:06:45.564256 19052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:45.563868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:39981 (local address 127.0.0.1:35710)
0807 07:06:45.564057 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:45.564127 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.1:39981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260807 07:06:45.642386 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 18798
I20260807 07:06:45.668295 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 18274
I20260807 07:06:45.669456 18363 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:06:45.780025 19052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:45.779522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.1:39981 (local address 127.0.0.1:35724)
0807 07:06:45.779738 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0807 07:06:45.779845 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.1:39981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260807 07:06:45.780030 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 18274
2026-08-07T07:06:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3183 ms)
[----------] 6 tests from EncryptionPolicyTest (23189 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786086078080656-21836-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 07:06:45 dist-test-slave-ncp9 krb5kdc[19060](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 07:06:45 dist-test-slave-ncp9 krb5kdc[19060](info): set up 2 sockets
Aug 07 07:06:45 dist-test-slave-ncp9 krb5kdc[19060](info): commencing operation
krb5kdc: starting...
W20260807 07:06:47.033919 21836 mini_kdc.cc:121] Time spent starting KDC: real 1.198s user 0.000s sys 0.057s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Aug 07 07:06:47 dist-test-slave-ncp9 krb5kdc[19060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T07:06:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:06:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786086078080656-21836-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786086078080656-21836-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 07:06:47.384334 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37889
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786086078080656-21836-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:06:47.726576 19082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:06:47.726922 19082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:06:47.727035 19082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:06:47.734375 19082 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:06:47.734478 19082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:06:47.734535 19082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:06:47.734597 19082 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:06:47.734642 19082 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:06:47.744390 19082 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37889
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786086078080656-21836-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:06:47.745554 19082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:06:47.746754 19082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:06:47.757934 19093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:47.757902 19096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:47.757968 19094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:06:47.758569 19082 server_base.cc:1034] running on GCE node
I20260807 07:06:47.759255 19082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:06:47.760517 19082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:06:47.761708 19082 hybrid_clock.cc:648] HybridClock initialized: now 1786086407761658 us; error 36 us; skew 500 ppm
I20260807 07:06:47.762027 19082 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:06:47 dist-test-slave-ncp9 krb5kdc[19060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:06:47.764997 19082 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 07:06:47.766319 19082 webserver.cc:460] Webserver started at http://127.0.0.1:42325/ using document root <none> and password file <none>
I20260807 07:06:47.766919 19082 fs_manager.cc:359] Metadata directory not provided
I20260807 07:06:47.767038 19082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:06:47.767318 19082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:06:47.770148 19082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "31c0524c50a24a7d91ff7717dd53d23c"
format_stamp: "Formatted at 2026-08-07 07:06:47 on dist-test-slave-ncp9"
server_key: "37d4481e971918b0c2a10acf19d3ec1b"
server_key_iv: "179c1ad5e3b1b625c7a1b993abbfbb4a"
server_key_version: "encryptionkey@0"
I20260807 07:06:47.770910 19082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "31c0524c50a24a7d91ff7717dd53d23c"
format_stamp: "Formatted at 2026-08-07 07:06:47 on dist-test-slave-ncp9"
server_key: "37d4481e971918b0c2a10acf19d3ec1b"
server_key_iv: "179c1ad5e3b1b625c7a1b993abbfbb4a"
server_key_version: "encryptionkey@0"
I20260807 07:06:47.775738 19082 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260807 07:06:47.778954 19106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:47.780184 19082 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260807 07:06:47.780349 19082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "31c0524c50a24a7d91ff7717dd53d23c"
format_stamp: "Formatted at 2026-08-07 07:06:47 on dist-test-slave-ncp9"
server_key: "37d4481e971918b0c2a10acf19d3ec1b"
server_key_iv: "179c1ad5e3b1b625c7a1b993abbfbb4a"
server_key_version: "encryptionkey@0"
I20260807 07:06:47.780525 19082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:06:47.793424 19082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:06:47.797399 19082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:06:47.797648 19082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:06:47.814803 19082 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37889
I20260807 07:06:47.814839 19176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37889 every 8 connection(s)
I20260807 07:06:47.816011 19082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:06:47.817840 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 19082
I20260807 07:06:47.818107 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:06:47.818922 21836 external_mini_cluster.cc:1428] Setting key 1dfe6234bd33329ae88b20e533f9c631
I20260807 07:06:47.820078 19178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 07:06:47 dist-test-slave-ncp9 krb5kdc[19060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260807 07:06:47.834427 19178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31c0524c50a24a7d91ff7717dd53d23c: Bootstrap starting.
I20260807 07:06:47.838001 19178 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31c0524c50a24a7d91ff7717dd53d23c: Neither blocks nor log segments found. Creating new log.
I20260807 07:06:47.839380 19178 log.cc:824] T 00000000000000000000000000000000 P 31c0524c50a24a7d91ff7717dd53d23c: Log is configured to *not* fsync() on all Append() calls
I20260807 07:06:47.842797 19187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:47.820726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55687 (local address 127.0.0.1:37889)
0807 07:06:47.821480 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:47.821490 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:47.821527 (+ 37us) server_negotiation.cc:407] Connection header received
0807 07:06:47.823075 (+ 1548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:47.823095 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:47.823440 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:47.823746 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:47.825575 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:47.826880 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:47.828670 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:47.829402 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:47.829538 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:47.834857 (+ 5319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:47.834867 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:47.834885 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:06:47.834935 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:47.838171 (+ 3236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:47.839258 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:47.839265 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:47.839272 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:47.839466 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:47.839976 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:47.839980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:47.839982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:47.840576 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:47.840933 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:47.841436 (+ 503us) server_negotiation.cc:299] Negotiation successful
0807 07:06:47.841791 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":379,"threads_started":1}
I20260807 07:06:47.842912 19178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31c0524c50a24a7d91ff7717dd53d23c: No bootstrap required, opened a new log
I20260807 07:06:47.846513 21836 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 07:06:47.852334 19178 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31c0524c50a24a7d91ff7717dd53d23c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c0524c50a24a7d91ff7717dd53d23c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37889 } }
I20260807 07:06:47.852712 19178 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31c0524c50a24a7d91ff7717dd53d23c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:06:47.852823 19178 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31c0524c50a24a7d91ff7717dd53d23c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31c0524c50a24a7d91ff7717dd53d23c, State: Initialized, Role: FOLLOWER
I20260807 07:06:47.853425 19178 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31c0524c50a24a7d91ff7717dd53d23c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c0524c50a24a7d91ff7717dd53d23c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37889 } }
I20260807 07:06:47.853657 19178 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31c0524c50a24a7d91ff7717dd53d23c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:06:47.853788 19178 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31c0524c50a24a7d91ff7717dd53d23c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:06:47.853945 19178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31c0524c50a24a7d91ff7717dd53d23c [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:06:47.856148 19178 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31c0524c50a24a7d91ff7717dd53d23c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c0524c50a24a7d91ff7717dd53d23c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37889 } }
I20260807 07:06:47.856606 19178 leader_election.cc:302] T 00000000000000000000000000000000 P 31c0524c50a24a7d91ff7717dd53d23c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31c0524c50a24a7d91ff7717dd53d23c; no voters:
I20260807 07:06:47.857216 19178 leader_election.cc:288] T 00000000000000000000000000000000 P 31c0524c50a24a7d91ff7717dd53d23c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 07:06:47.857335 19199 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31c0524c50a24a7d91ff7717dd53d23c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:06:47.859424 19178 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31c0524c50a24a7d91ff7717dd53d23c [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:06:47.861037 19199 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31c0524c50a24a7d91ff7717dd53d23c [term 1 LEADER]: Becoming Leader. State: Replica: 31c0524c50a24a7d91ff7717dd53d23c, State: Running, Role: LEADER
I20260807 07:06:47.861635 19199 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31c0524c50a24a7d91ff7717dd53d23c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c0524c50a24a7d91ff7717dd53d23c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37889 } }
I20260807 07:06:47.864442 19201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31c0524c50a24a7d91ff7717dd53d23c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31c0524c50a24a7d91ff7717dd53d23c. Latest consensus state: current_term: 1 leader_uuid: "31c0524c50a24a7d91ff7717dd53d23c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c0524c50a24a7d91ff7717dd53d23c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37889 } } }
I20260807 07:06:47.864823 19201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31c0524c50a24a7d91ff7717dd53d23c [sys.catalog]: This master's current role is: LEADER
I20260807 07:06:47.865334 19200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31c0524c50a24a7d91ff7717dd53d23c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31c0524c50a24a7d91ff7717dd53d23c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c0524c50a24a7d91ff7717dd53d23c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37889 } } }
I20260807 07:06:47.865608 19200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31c0524c50a24a7d91ff7717dd53d23c [sys.catalog]: This master's current role is: LEADER
I20260807 07:06:47.867616 19187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:47.851973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54171 (local address 127.0.0.1:37889)
0807 07:06:47.852152 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:47.852158 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:47.852607 (+ 449us) server_negotiation.cc:407] Connection header received
0807 07:06:47.852756 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:47.852761 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:47.852833 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:47.852950 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:47.854454 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:47.855273 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:47.856510 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:47.857062 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:47.857192 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:47.859722 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:47.859728 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:47.859733 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:06:47.859762 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:47.863611 (+ 3849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:47.864681 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:47.864688 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:47.864692 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:47.864959 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:47.865587 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:47.865596 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:47.865600 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:47.865994 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:47.866319 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:47.867035 (+ 716us) server_negotiation.cc:299] Negotiation successful
0807 07:06:47.867239 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 07:06:47.877764 19210 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:06:47.884176 19210 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:06:47.893905 19210 catalog_manager.cc:1347] Generated new cluster ID: 8399df08f12e48e3b91f3e90dbc1bac3
I20260807 07:06:47.894083 19210 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:06:47.904482 19210 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:06:47.905810 19210 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:06:47.913869 19210 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31c0524c50a24a7d91ff7717dd53d23c: Generated new TSK 0
I20260807 07:06:47.914628 19210 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 07:06:47.928843 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 19082
I20260807 07:06:47.930006 19171 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:06:48.021739 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 19082
W20260807 07:06:48.043956 19058 connection.cc:535] client connection to 127.0.0.1:37889 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37889): Connection reset by peer (error 104)
2026-08-07T07:06:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2227 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786086078080656-21836-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 07:06:48 dist-test-slave-ncp9 krb5kdc[19228](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 07:06:48 dist-test-slave-ncp9 krb5kdc[19228](info): set up 2 sockets
Aug 07 07:06:48 dist-test-slave-ncp9 krb5kdc[19228](info): commencing operation
krb5kdc: starting...
W20260807 07:06:49.256982 21836 mini_kdc.cc:121] Time spent starting KDC: real 1.194s user 0.004s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Aug 07 07:06:49 dist-test-slave-ncp9 krb5kdc[19228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T07:06:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:06:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786086078080656-21836-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786086078080656-21836-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 07:06:49.623447 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39885
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786086078080656-21836-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:06:49.969482 19249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:06:49.969806 19249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:06:49.969902 19249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:06:49.977284 19249 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:06:49.977406 19249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:06:49.977452 19249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:06:49.977540 19249 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:06:49.977586 19249 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:06:49.987576 19249 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39885
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786086078080656-21836-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:06:49.988883 19249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:06:49.990167 19249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:06:50.001816 19262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:50.001747 19266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:50.001794 19264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:06:50.002631 19249 server_base.cc:1034] running on GCE node
I20260807 07:06:50.003417 19249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:06:50.004848 19249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:06:50.006089 19249 hybrid_clock.cc:648] HybridClock initialized: now 1786086410006026 us; error 55 us; skew 500 ppm
I20260807 07:06:50.006449 19249 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:06:50 dist-test-slave-ncp9 krb5kdc[19228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:06:50.009649 19249 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 07:06:50.011073 19249 webserver.cc:460] Webserver started at http://127.0.0.1:41027/ using document root <none> and password file <none>
I20260807 07:06:50.011755 19249 fs_manager.cc:359] Metadata directory not provided
I20260807 07:06:50.011883 19249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:06:50.012181 19249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:06:50.015261 19249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "4cba1872937b4f87b11dfa468120eaa5"
format_stamp: "Formatted at 2026-08-07 07:06:50 on dist-test-slave-ncp9"
server_key: "ac010b87e839b7d093203ef4dd1d5dfd"
server_key_iv: "85286624cf34e054877247e434191300"
server_key_version: "encryptionkey@0"
I20260807 07:06:50.016032 19249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "4cba1872937b4f87b11dfa468120eaa5"
format_stamp: "Formatted at 2026-08-07 07:06:50 on dist-test-slave-ncp9"
server_key: "ac010b87e839b7d093203ef4dd1d5dfd"
server_key_iv: "85286624cf34e054877247e434191300"
server_key_version: "encryptionkey@0"
I20260807 07:06:50.021293 19249 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260807 07:06:50.025027 19276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:50.026309 19249 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 07:06:50.026476 19249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "4cba1872937b4f87b11dfa468120eaa5"
format_stamp: "Formatted at 2026-08-07 07:06:50 on dist-test-slave-ncp9"
server_key: "ac010b87e839b7d093203ef4dd1d5dfd"
server_key_iv: "85286624cf34e054877247e434191300"
server_key_version: "encryptionkey@0"
I20260807 07:06:50.026669 19249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:06:50.048108 19249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:06:50.052847 19249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:06:50.053200 19249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:06:50.071631 19249 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39885
I20260807 07:06:50.071674 19353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39885 every 8 connection(s)
I20260807 07:06:50.073031 19249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:06:50.077172 19354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:06:50.079173 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 19249
I20260807 07:06:50.079352 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:06:50.079800 21836 external_mini_cluster.cc:1428] Setting key 862b21adc2139dfab90a14def73777d7
I20260807 07:06:50.090421 19354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cba1872937b4f87b11dfa468120eaa5: Bootstrap starting.
I20260807 07:06:50.093832 19354 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4cba1872937b4f87b11dfa468120eaa5: Neither blocks nor log segments found. Creating new log.
I20260807 07:06:50.094954 19354 log.cc:824] T 00000000000000000000000000000000 P 4cba1872937b4f87b11dfa468120eaa5: Log is configured to *not* fsync() on all Append() calls
I20260807 07:06:50.095002 19357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:50.081731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40551 (local address 127.0.0.1:39885)
0807 07:06:50.082582 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:50.082597 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:50.082647 (+ 50us) server_negotiation.cc:407] Connection header received
0807 07:06:50.084980 (+ 2333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:50.085006 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:50.085487 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:50.085902 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:50.086431 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:50.088495 (+ 2064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:50.090007 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:50.091046 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:50.091248 (+ 202us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:50.092622 (+ 1374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:50.092630 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:50.092644 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:06:50.093304 (+ 660us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:50.093389 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:06:50.093415 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:50.093561 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:50.093853 (+ 292us) server_negotiation.cc:299] Negotiation successful
0807 07:06:50.094035 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":364,"threads_started":1}
I20260807 07:06:50.098346 19354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cba1872937b4f87b11dfa468120eaa5: No bootstrap required, opened a new log
I20260807 07:06:50.099272 21836 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 07:06:50.107072 19354 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4cba1872937b4f87b11dfa468120eaa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cba1872937b4f87b11dfa468120eaa5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39885 } }
I20260807 07:06:50.107471 19354 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4cba1872937b4f87b11dfa468120eaa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:06:50.107584 19354 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4cba1872937b4f87b11dfa468120eaa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cba1872937b4f87b11dfa468120eaa5, State: Initialized, Role: FOLLOWER
I20260807 07:06:50.108153 19354 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4cba1872937b4f87b11dfa468120eaa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cba1872937b4f87b11dfa468120eaa5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39885 } }
I20260807 07:06:50.108453 19354 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4cba1872937b4f87b11dfa468120eaa5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:06:50.108619 19354 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4cba1872937b4f87b11dfa468120eaa5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:06:50.108803 19354 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4cba1872937b4f87b11dfa468120eaa5 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:06:50.110281 19357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:50.103584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60509 (local address 127.0.0.1:39885)
0807 07:06:50.103771 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:50.103778 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:50.104114 (+ 336us) server_negotiation.cc:407] Connection header received
0807 07:06:50.104272 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:50.104277 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:50.104352 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:50.104455 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:50.104953 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:50.106108 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:50.107494 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:50.108084 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:50.108189 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:50.109493 (+ 1304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:50.109498 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:50.109501 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:06:50.109519 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:50.109582 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:06:50.109601 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:50.109723 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:50.109919 (+ 196us) server_negotiation.cc:299] Negotiation successful
0807 07:06:50.109982 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 07:06:50.111605 19354 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4cba1872937b4f87b11dfa468120eaa5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cba1872937b4f87b11dfa468120eaa5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39885 } }
I20260807 07:06:50.112190 19354 leader_election.cc:302] T 00000000000000000000000000000000 P 4cba1872937b4f87b11dfa468120eaa5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cba1872937b4f87b11dfa468120eaa5; no voters:
I20260807 07:06:50.112787 19354 leader_election.cc:288] T 00000000000000000000000000000000 P 4cba1872937b4f87b11dfa468120eaa5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 07:06:50.112931 19373 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4cba1872937b4f87b11dfa468120eaa5 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:06:50.113629 19373 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4cba1872937b4f87b11dfa468120eaa5 [term 1 LEADER]: Becoming Leader. State: Replica: 4cba1872937b4f87b11dfa468120eaa5, State: Running, Role: LEADER
I20260807 07:06:50.114154 19373 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4cba1872937b4f87b11dfa468120eaa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cba1872937b4f87b11dfa468120eaa5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39885 } }
I20260807 07:06:50.115309 19354 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4cba1872937b4f87b11dfa468120eaa5 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:06:50.131322 19375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cba1872937b4f87b11dfa468120eaa5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4cba1872937b4f87b11dfa468120eaa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cba1872937b4f87b11dfa468120eaa5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39885 } } }
I20260807 07:06:50.131868 19375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cba1872937b4f87b11dfa468120eaa5 [sys.catalog]: This master's current role is: LEADER
I20260807 07:06:50.132350 19376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cba1872937b4f87b11dfa468120eaa5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4cba1872937b4f87b11dfa468120eaa5. Latest consensus state: current_term: 1 leader_uuid: "4cba1872937b4f87b11dfa468120eaa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cba1872937b4f87b11dfa468120eaa5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39885 } } }
I20260807 07:06:50.132763 19376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cba1872937b4f87b11dfa468120eaa5 [sys.catalog]: This master's current role is: LEADER
I20260807 07:06:50.133365 19394 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:06:50.138814 19394 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:06:50.148759 19394 catalog_manager.cc:1347] Generated new cluster ID: 0f802465dca14340bbc5992fc23e4b7d
I20260807 07:06:50.148874 19394 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:06:50.161729 19394 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:06:50.163280 19394 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:06:50.184970 19394 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4cba1872937b4f87b11dfa468120eaa5: Generated new TSK 0
I20260807 07:06:50.185913 19394 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 07:06:50.195479 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 19249
W20260807 07:06:50.197048 19347 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40551
W20260807 07:06:50.197916 21836 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 19249: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 07:06:50.198083 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 19249
W20260807 07:06:50.222671 19225 connection.cc:535] client connection to 127.0.0.1:39885 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39885): Connection reset by peer (error 104)
2026-08-07T07:06:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2178 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786086078080656-21836-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 07:06:50 dist-test-slave-ncp9 krb5kdc[19410](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 07:06:50 dist-test-slave-ncp9 krb5kdc[19410](info): set up 2 sockets
Aug 07 07:06:50 dist-test-slave-ncp9 krb5kdc[19410](info): commencing operation
krb5kdc: starting...
W20260807 07:06:51.446302 21836 mini_kdc.cc:121] Time spent starting KDC: real 1.204s user 0.009s sys 0.055s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Aug 07 07:06:51 dist-test-slave-ncp9 krb5kdc[19410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T07:06:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:06:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786086078080656-21836-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786086078080656-21836-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 07:06:51.808594 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33427
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786086078080656-21836-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33427
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:06:52.162847 19427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:06:52.163199 19427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:06:52.163344 19427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:06:52.170749 19427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:06:52.170893 19427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:06:52.170975 19427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:06:52.171057 19427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:06:52.171119 19427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:06:52.181335 19427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33427
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786086078080656-21836-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:06:52.182893 19427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:06:52.184185 19427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:06:52.195470 19438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:52.195412 19440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:52.195858 19436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:06:52.196213 19427 server_base.cc:1034] running on GCE node
I20260807 07:06:52.196925 19427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:06:52.198273 19427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:06:52.199463 19427 hybrid_clock.cc:648] HybridClock initialized: now 1786086412199403 us; error 57 us; skew 500 ppm
I20260807 07:06:52.199795 19427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:06:52 dist-test-slave-ncp9 krb5kdc[19410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:06:52.202971 19427 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 07:06:52.204385 19427 webserver.cc:460] Webserver started at http://127.0.0.1:45653/ using document root <none> and password file <none>
I20260807 07:06:52.205070 19427 fs_manager.cc:359] Metadata directory not provided
I20260807 07:06:52.205189 19427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:06:52.205477 19427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:06:52.208341 19427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "fffeaa8c300d492eb8036f91eb4f80b0"
format_stamp: "Formatted at 2026-08-07 07:06:52 on dist-test-slave-ncp9"
server_key: "0a3453cccea7f0402233739cc7c743ee"
server_key_iv: "d409f15114cf4dc75bc84c3e2ac7723d"
server_key_version: "encryptionkey@0"
I20260807 07:06:52.209173 19427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "fffeaa8c300d492eb8036f91eb4f80b0"
format_stamp: "Formatted at 2026-08-07 07:06:52 on dist-test-slave-ncp9"
server_key: "0a3453cccea7f0402233739cc7c743ee"
server_key_iv: "d409f15114cf4dc75bc84c3e2ac7723d"
server_key_version: "encryptionkey@0"
I20260807 07:06:52.214088 19427 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 07:06:52.217543 19448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:52.218777 19427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 07:06:52.218930 19427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "fffeaa8c300d492eb8036f91eb4f80b0"
format_stamp: "Formatted at 2026-08-07 07:06:52 on dist-test-slave-ncp9"
server_key: "0a3453cccea7f0402233739cc7c743ee"
server_key_iv: "d409f15114cf4dc75bc84c3e2ac7723d"
server_key_version: "encryptionkey@0"
I20260807 07:06:52.219115 19427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:06:52.224757 19427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:06:52.228490 19427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:06:52.228752 19427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:06:52.246187 19427 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33427
I20260807 07:06:52.246225 19532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33427 every 8 connection(s)
I20260807 07:06:52.247563 19427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:06:52.251466 19533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:06:52.255664 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 19427
I20260807 07:06:52.255887 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:06:52.256424 21836 external_mini_cluster.cc:1428] Setting key 201e79e6e48dda6a081959b6eded69c4
I20260807 07:06:52.264671 19538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:52.258297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60151 (local address 127.0.0.1:33427)
0807 07:06:52.259165 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:52.259177 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:52.259221 (+ 44us) server_negotiation.cc:407] Connection header received
0807 07:06:52.260986 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:52.261013 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:52.261426 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:52.261800 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:52.262983 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:52.262991 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:52.263005 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:06:52.263024 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:52.263100 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:06:52.263128 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:52.263309 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:52.263432 (+ 123us) server_negotiation.cc:299] Negotiation successful
0807 07:06:52.263663 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":363,"threads_started":1}
I20260807 07:06:52.265262 19533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fffeaa8c300d492eb8036f91eb4f80b0: Bootstrap starting.
I20260807 07:06:52.269023 21836 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 07:06:52.269060 19533 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fffeaa8c300d492eb8036f91eb4f80b0: Neither blocks nor log segments found. Creating new log.
I20260807 07:06:52.270087 19533 log.cc:824] T 00000000000000000000000000000000 P fffeaa8c300d492eb8036f91eb4f80b0: Log is configured to *not* fsync() on all Append() calls
I20260807 07:06:52.273144 19533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fffeaa8c300d492eb8036f91eb4f80b0: No bootstrap required, opened a new log
I20260807 07:06:52.275784 19538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:52.273260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35507 (local address 127.0.0.1:33427)
0807 07:06:52.273431 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:52.273438 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:52.273828 (+ 390us) server_negotiation.cc:407] Connection header received
0807 07:06:52.273961 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:52.273965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:52.274026 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:52.274139 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:52.275042 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:52.275052 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:52.275057 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:06:52.275083 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:52.275133 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:06:52.275154 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:52.275279 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:52.275429 (+ 150us) server_negotiation.cc:299] Negotiation successful
0807 07:06:52.275489 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260807 07:06:52.281561 19533 raft_consensus.cc:348] T 00000000000000000000000000000000 P fffeaa8c300d492eb8036f91eb4f80b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffeaa8c300d492eb8036f91eb4f80b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33427 } }
I20260807 07:06:52.281952 19533 raft_consensus.cc:374] T 00000000000000000000000000000000 P fffeaa8c300d492eb8036f91eb4f80b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:06:52.282107 19533 raft_consensus.cc:729] T 00000000000000000000000000000000 P fffeaa8c300d492eb8036f91eb4f80b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fffeaa8c300d492eb8036f91eb4f80b0, State: Initialized, Role: FOLLOWER
I20260807 07:06:52.282732 19533 consensus_queue.cc:260] T 00000000000000000000000000000000 P fffeaa8c300d492eb8036f91eb4f80b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffeaa8c300d492eb8036f91eb4f80b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33427 } }
I20260807 07:06:52.282970 19533 raft_consensus.cc:388] T 00000000000000000000000000000000 P fffeaa8c300d492eb8036f91eb4f80b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:06:52.283110 19533 raft_consensus.cc:482] T 00000000000000000000000000000000 P fffeaa8c300d492eb8036f91eb4f80b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:06:52.283265 19533 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fffeaa8c300d492eb8036f91eb4f80b0 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:06:52.285588 19533 raft_consensus.cc:504] T 00000000000000000000000000000000 P fffeaa8c300d492eb8036f91eb4f80b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffeaa8c300d492eb8036f91eb4f80b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33427 } }
I20260807 07:06:52.286070 19533 leader_election.cc:302] T 00000000000000000000000000000000 P fffeaa8c300d492eb8036f91eb4f80b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fffeaa8c300d492eb8036f91eb4f80b0; no voters:
I20260807 07:06:52.286629 19533 leader_election.cc:288] T 00000000000000000000000000000000 P fffeaa8c300d492eb8036f91eb4f80b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 07:06:52.286761 19552 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fffeaa8c300d492eb8036f91eb4f80b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:06:52.287412 19552 raft_consensus.cc:686] T 00000000000000000000000000000000 P fffeaa8c300d492eb8036f91eb4f80b0 [term 1 LEADER]: Becoming Leader. State: Replica: fffeaa8c300d492eb8036f91eb4f80b0, State: Running, Role: LEADER
I20260807 07:06:52.287954 19552 consensus_queue.cc:237] T 00000000000000000000000000000000 P fffeaa8c300d492eb8036f91eb4f80b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffeaa8c300d492eb8036f91eb4f80b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33427 } }
I20260807 07:06:52.288650 19533 sys_catalog.cc:564] T 00000000000000000000000000000000 P fffeaa8c300d492eb8036f91eb4f80b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:06:52.291268 19554 sys_catalog.cc:455] T 00000000000000000000000000000000 P fffeaa8c300d492eb8036f91eb4f80b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fffeaa8c300d492eb8036f91eb4f80b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffeaa8c300d492eb8036f91eb4f80b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33427 } } }
I20260807 07:06:52.291231 19555 sys_catalog.cc:455] T 00000000000000000000000000000000 P fffeaa8c300d492eb8036f91eb4f80b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fffeaa8c300d492eb8036f91eb4f80b0. Latest consensus state: current_term: 1 leader_uuid: "fffeaa8c300d492eb8036f91eb4f80b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffeaa8c300d492eb8036f91eb4f80b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33427 } } }
I20260807 07:06:52.291715 19554 sys_catalog.cc:458] T 00000000000000000000000000000000 P fffeaa8c300d492eb8036f91eb4f80b0 [sys.catalog]: This master's current role is: LEADER
I20260807 07:06:52.291718 19555 sys_catalog.cc:458] T 00000000000000000000000000000000 P fffeaa8c300d492eb8036f91eb4f80b0 [sys.catalog]: This master's current role is: LEADER
I20260807 07:06:52.292626 19566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:06:52.297996 19566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:06:52.307067 19566 catalog_manager.cc:1347] Generated new cluster ID: 3d316f3c663748afa86344cca24db2d0
I20260807 07:06:52.307194 19566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:06:52.324632 19566 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:06:52.325080 19566 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:06:52.335237 19566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fffeaa8c300d492eb8036f91eb4f80b0: Generated new TSK 0
I20260807 07:06:52.335981 19566 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 07:06:52.347251 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 19427
W20260807 07:06:52.348340 19527 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60151
W20260807 07:06:52.349035 21836 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 19427: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 07:06:52.349167 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 19427
2026-08-07T07:06:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2148 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786086078080656-21836-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 07:06:52 dist-test-slave-ncp9 krb5kdc[19584](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 07:06:52 dist-test-slave-ncp9 krb5kdc[19584](info): set up 2 sockets
Aug 07 07:06:52 dist-test-slave-ncp9 krb5kdc[19584](info): commencing operation
krb5kdc: starting...
W20260807 07:06:53.587337 21836 mini_kdc.cc:121] Time spent starting KDC: real 1.197s user 0.000s sys 0.055s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Aug 07 07:06:53 dist-test-slave-ncp9 krb5kdc[19584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T07:06:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:06:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786086078080656-21836-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786086078080656-21836-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 07:06:53.939200 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37409
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786086078080656-21836-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37409
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:06:54.284550 19602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:06:54.284862 19602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:06:54.285027 19602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:06:54.292368 19602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:06:54.292495 19602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:06:54.292577 19602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:06:54.292661 19602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:06:54.292719 19602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:06:54.302760 19602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786086078080656-21836-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:06:54.304116 19602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:06:54.305529 19602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:06:54.318003 19608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:54.318024 19609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:54.318177 19611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:06:54.318828 19602 server_base.cc:1034] running on GCE node
I20260807 07:06:54.319654 19602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:06:54.321132 19602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:06:54.322335 19602 hybrid_clock.cc:648] HybridClock initialized: now 1786086414322287 us; error 41 us; skew 500 ppm
I20260807 07:06:54.322695 19602 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:06:54 dist-test-slave-ncp9 krb5kdc[19584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086414, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:06:54.330204 19602 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 07:06:54.331634 19602 webserver.cc:460] Webserver started at http://127.0.0.1:39691/ using document root <none> and password file <none>
I20260807 07:06:54.332243 19602 fs_manager.cc:359] Metadata directory not provided
I20260807 07:06:54.332356 19602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:06:54.332617 19602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:06:54.335485 19602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "087e5d8084484293b8998037ec09029e"
format_stamp: "Formatted at 2026-08-07 07:06:54 on dist-test-slave-ncp9"
server_key: "c282b0e92d768168d48e005b27fa687e"
server_key_iv: "c554c9071dcc1930d573c66d0e629276"
server_key_version: "encryptionkey@0"
I20260807 07:06:54.336189 19602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "087e5d8084484293b8998037ec09029e"
format_stamp: "Formatted at 2026-08-07 07:06:54 on dist-test-slave-ncp9"
server_key: "c282b0e92d768168d48e005b27fa687e"
server_key_iv: "c554c9071dcc1930d573c66d0e629276"
server_key_version: "encryptionkey@0"
I20260807 07:06:54.341068 19602 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260807 07:06:54.344307 19622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:54.345710 19602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 07:06:54.345862 19602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "087e5d8084484293b8998037ec09029e"
format_stamp: "Formatted at 2026-08-07 07:06:54 on dist-test-slave-ncp9"
server_key: "c282b0e92d768168d48e005b27fa687e"
server_key_iv: "c554c9071dcc1930d573c66d0e629276"
server_key_version: "encryptionkey@0"
I20260807 07:06:54.346036 19602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:06:54.357448 19602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:06:54.361712 19602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:06:54.361975 19602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:06:54.381104 19602 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37409
I20260807 07:06:54.381126 19699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37409 every 8 connection(s)
I20260807 07:06:54.382548 19602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:06:54.384403 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 19602
I20260807 07:06:54.384598 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:06:54.385215 21836 external_mini_cluster.cc:1428] Setting key e8a89ac3075cab42fea42a710dd04254
I20260807 07:06:54.387081 19701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 07:06:54 dist-test-slave-ncp9 krb5kdc[19584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260807 07:06:54.400982 19701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 087e5d8084484293b8998037ec09029e: Bootstrap starting.
I20260807 07:06:54.404879 19701 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 087e5d8084484293b8998037ec09029e: Neither blocks nor log segments found. Creating new log.
I20260807 07:06:54.406101 19701 log.cc:824] T 00000000000000000000000000000000 P 087e5d8084484293b8998037ec09029e: Log is configured to *not* fsync() on all Append() calls
I20260807 07:06:54.409348 19701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 087e5d8084484293b8998037ec09029e: No bootstrap required, opened a new log
I20260807 07:06:54.410300 19708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:54.386959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41261 (local address 127.0.0.1:37409)
0807 07:06:54.387786 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:54.387798 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:54.387837 (+ 39us) server_negotiation.cc:407] Connection header received
0807 07:06:54.389536 (+ 1699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:54.389566 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:54.389953 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:54.390292 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:54.392137 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:54.393565 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:54.395367 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:54.396060 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:54.396196 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:54.401989 (+ 5793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:54.401999 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:54.402015 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:06:54.402062 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:54.405726 (+ 3664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:54.406656 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:54.406664 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:54.406671 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:54.406900 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:54.407329 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:54.407333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:54.407335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:54.407918 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:54.408214 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:54.408734 (+ 520us) server_negotiation.cc:299] Negotiation successful
0807 07:06:54.409210 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":378,"threads_started":1}
I20260807 07:06:54.414340 21836 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 07:06:54.417714 19701 raft_consensus.cc:348] T 00000000000000000000000000000000 P 087e5d8084484293b8998037ec09029e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "087e5d8084484293b8998037ec09029e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37409 } }
I20260807 07:06:54.418144 19701 raft_consensus.cc:374] T 00000000000000000000000000000000 P 087e5d8084484293b8998037ec09029e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:06:54.418246 19701 raft_consensus.cc:729] T 00000000000000000000000000000000 P 087e5d8084484293b8998037ec09029e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 087e5d8084484293b8998037ec09029e, State: Initialized, Role: FOLLOWER
I20260807 07:06:54.418938 19701 consensus_queue.cc:260] T 00000000000000000000000000000000 P 087e5d8084484293b8998037ec09029e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "087e5d8084484293b8998037ec09029e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37409 } }
I20260807 07:06:54.419188 19701 raft_consensus.cc:388] T 00000000000000000000000000000000 P 087e5d8084484293b8998037ec09029e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:06:54.419327 19701 raft_consensus.cc:482] T 00000000000000000000000000000000 P 087e5d8084484293b8998037ec09029e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:06:54.419492 19701 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 087e5d8084484293b8998037ec09029e [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:06:54.422216 19701 raft_consensus.cc:504] T 00000000000000000000000000000000 P 087e5d8084484293b8998037ec09029e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "087e5d8084484293b8998037ec09029e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37409 } }
I20260807 07:06:54.422875 19701 leader_election.cc:302] T 00000000000000000000000000000000 P 087e5d8084484293b8998037ec09029e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 087e5d8084484293b8998037ec09029e; no voters:
I20260807 07:06:54.423619 19701 leader_election.cc:288] T 00000000000000000000000000000000 P 087e5d8084484293b8998037ec09029e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 07:06:54.423715 19715 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 087e5d8084484293b8998037ec09029e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:06:54.425551 19715 raft_consensus.cc:686] T 00000000000000000000000000000000 P 087e5d8084484293b8998037ec09029e [term 1 LEADER]: Becoming Leader. State: Replica: 087e5d8084484293b8998037ec09029e, State: Running, Role: LEADER
I20260807 07:06:54.426115 19701 sys_catalog.cc:564] T 00000000000000000000000000000000 P 087e5d8084484293b8998037ec09029e [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:06:54.426156 19715 consensus_queue.cc:237] T 00000000000000000000000000000000 P 087e5d8084484293b8998037ec09029e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "087e5d8084484293b8998037ec09029e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37409 } }
I20260807 07:06:54.431080 19717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 087e5d8084484293b8998037ec09029e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 087e5d8084484293b8998037ec09029e. Latest consensus state: current_term: 1 leader_uuid: "087e5d8084484293b8998037ec09029e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "087e5d8084484293b8998037ec09029e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37409 } } }
I20260807 07:06:54.431608 19717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 087e5d8084484293b8998037ec09029e [sys.catalog]: This master's current role is: LEADER
I20260807 07:06:54.436012 19716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 087e5d8084484293b8998037ec09029e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "087e5d8084484293b8998037ec09029e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "087e5d8084484293b8998037ec09029e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37409 } } }
I20260807 07:06:54.436964 19716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 087e5d8084484293b8998037ec09029e [sys.catalog]: This master's current role is: LEADER
I20260807 07:06:54.440141 19708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:54.418852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48133 (local address 127.0.0.1:37409)
0807 07:06:54.419098 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:54.419106 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:54.421477 (+ 2371us) server_negotiation.cc:407] Connection header received
0807 07:06:54.421645 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:54.421650 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:54.421715 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:54.421798 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:54.423775 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:54.425130 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:54.426886 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:54.427738 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:54.427886 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:54.431435 (+ 3549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:54.431443 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:54.431449 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:06:54.431495 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:54.435731 (+ 4236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:54.436675 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:54.436680 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:54.436684 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:54.436883 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:06:54.437434 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:06:54.437440 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:06:54.437444 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:06:54.437876 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:54.438159 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:54.439440 (+ 1281us) server_negotiation.cc:299] Negotiation successful
0807 07:06:54.439695 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":71}
I20260807 07:06:54.442028 19730 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:06:54.447309 19730 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:06:54.456943 19730 catalog_manager.cc:1347] Generated new cluster ID: 6d7e31cf93a048b191678b9b80563e13
I20260807 07:06:54.457077 19730 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:06:54.474206 19730 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:06:54.475639 19730 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:06:54.486822 19730 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 087e5d8084484293b8998037ec09029e: Generated new TSK 0
I20260807 07:06:54.487737 19730 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 07:06:54.499217 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 19602
I20260807 07:06:54.500466 19688 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:06:54.592417 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 19602
2026-08-07T07:06:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2245 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786086078080656-21836-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 07:06:54 dist-test-slave-ncp9 krb5kdc[19743](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 07:06:54 dist-test-slave-ncp9 krb5kdc[19743](info): set up 2 sockets
Aug 07 07:06:54 dist-test-slave-ncp9 krb5kdc[19743](info): commencing operation
krb5kdc: starting...
W20260807 07:06:55.831755 21836 mini_kdc.cc:121] Time spent starting KDC: real 1.197s user 0.000s sys 0.060s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Aug 07 07:06:56 dist-test-slave-ncp9 krb5kdc[19743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T07:06:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:06:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786086078080656-21836-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786086078080656-21836-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 07:06:56.196290 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42719
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786086078080656-21836-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:06:56.551611 19765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:06:56.551913 19765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:06:56.552049 19765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:06:56.559486 19765 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:06:56.559623 19765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:06:56.559676 19765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:06:56.559777 19765 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:06:56.559830 19765 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:06:56.570500 19765 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786086078080656-21836-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:06:56.571898 19765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:06:56.573320 19765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:06:56.586256 19773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:56.586381 19774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:56.586403 19778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:06:56.587201 19765 server_base.cc:1034] running on GCE node
I20260807 07:06:56.588099 19765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:06:56.589690 19765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:06:56.590900 19765 hybrid_clock.cc:648] HybridClock initialized: now 1786086416590855 us; error 44 us; skew 500 ppm
I20260807 07:06:56.591257 19765 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:06:56 dist-test-slave-ncp9 krb5kdc[19743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:06:56.594475 19765 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 07:06:56.596004 19765 webserver.cc:460] Webserver started at http://127.0.0.1:46543/ using document root <none> and password file <none>
I20260807 07:06:56.596678 19765 fs_manager.cc:359] Metadata directory not provided
I20260807 07:06:56.596799 19765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:06:56.597138 19765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:06:56.600199 19765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "8416e2af8b3b474e9f464633554c2f91"
format_stamp: "Formatted at 2026-08-07 07:06:56 on dist-test-slave-ncp9"
server_key: "e692ebac11fa16c2def9a71286ade564"
server_key_iv: "3cb141aa17cf4a98e39c04d0d9226d0f"
server_key_version: "encryptionkey@0"
I20260807 07:06:56.601009 19765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "8416e2af8b3b474e9f464633554c2f91"
format_stamp: "Formatted at 2026-08-07 07:06:56 on dist-test-slave-ncp9"
server_key: "e692ebac11fa16c2def9a71286ade564"
server_key_iv: "3cb141aa17cf4a98e39c04d0d9226d0f"
server_key_version: "encryptionkey@0"
I20260807 07:06:56.607205 19765 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260807 07:06:56.610620 19790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:56.611866 19765 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 07:06:56.612025 19765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "8416e2af8b3b474e9f464633554c2f91"
format_stamp: "Formatted at 2026-08-07 07:06:56 on dist-test-slave-ncp9"
server_key: "e692ebac11fa16c2def9a71286ade564"
server_key_iv: "3cb141aa17cf4a98e39c04d0d9226d0f"
server_key_version: "encryptionkey@0"
I20260807 07:06:56.612210 19765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:06:56.643191 19765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:06:56.649066 19765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:06:56.649374 19765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:06:56.667785 19765 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42719
I20260807 07:06:56.667819 19864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42719 every 8 connection(s)
I20260807 07:06:56.669139 19765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:06:56.670367 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 19765
I20260807 07:06:56.670539 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:06:56.671093 21836 external_mini_cluster.cc:1428] Setting key ccb8c1863bd03ce8f4d38d38ac87cf4e
I20260807 07:06:56.673564 19865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:06:56.684656 19868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:56.672861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46277 (local address 127.0.0.1:42719)
0807 07:06:56.673651 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:56.673666 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:56.673725 (+ 59us) server_negotiation.cc:407] Connection header received
0807 07:06:56.675336 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:56.675356 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:56.675726 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:56.676041 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:56.676633 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:56.678030 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:56.679819 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:56.680471 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:56.680605 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:56.681802 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:56.681810 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:56.681822 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:06:56.682430 (+ 608us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:56.682511 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:06:56.682538 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:56.682633 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:56.683436 (+ 803us) server_negotiation.cc:299] Negotiation successful
0807 07:06:56.683682 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":336,"threads_started":1}
I20260807 07:06:56.685824 19865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8416e2af8b3b474e9f464633554c2f91: Bootstrap starting.
I20260807 07:06:56.688325 21836 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 07:06:56.689137 19865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8416e2af8b3b474e9f464633554c2f91: Neither blocks nor log segments found. Creating new log.
I20260807 07:06:56.690140 19865 log.cc:824] T 00000000000000000000000000000000 P 8416e2af8b3b474e9f464633554c2f91: Log is configured to *not* fsync() on all Append() calls
I20260807 07:06:56.693527 19865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8416e2af8b3b474e9f464633554c2f91: No bootstrap required, opened a new log
I20260807 07:06:56.699142 19868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:56.693110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46499 (local address 127.0.0.1:42719)
0807 07:06:56.693262 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:56.693268 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:56.693733 (+ 465us) server_negotiation.cc:407] Connection header received
0807 07:06:56.693926 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:56.693930 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:56.693997 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:56.694086 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:56.694629 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:56.695398 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:56.696736 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:06:56.697252 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:06:56.697362 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:06:56.698412 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:56.698416 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:56.698420 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:06:56.698440 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:56.698479 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:06:56.698498 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:56.698586 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:56.698730 (+ 144us) server_negotiation.cc:299] Negotiation successful
0807 07:06:56.698830 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260807 07:06:56.701836 19865 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8416e2af8b3b474e9f464633554c2f91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8416e2af8b3b474e9f464633554c2f91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42719 } }
I20260807 07:06:56.702227 19865 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8416e2af8b3b474e9f464633554c2f91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:06:56.702325 19865 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8416e2af8b3b474e9f464633554c2f91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8416e2af8b3b474e9f464633554c2f91, State: Initialized, Role: FOLLOWER
I20260807 07:06:56.702913 19865 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8416e2af8b3b474e9f464633554c2f91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8416e2af8b3b474e9f464633554c2f91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42719 } }
I20260807 07:06:56.703137 19865 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8416e2af8b3b474e9f464633554c2f91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:06:56.703274 19865 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8416e2af8b3b474e9f464633554c2f91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:06:56.703441 19865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8416e2af8b3b474e9f464633554c2f91 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:06:56.705791 19865 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8416e2af8b3b474e9f464633554c2f91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8416e2af8b3b474e9f464633554c2f91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42719 } }
I20260807 07:06:56.706358 19865 leader_election.cc:302] T 00000000000000000000000000000000 P 8416e2af8b3b474e9f464633554c2f91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8416e2af8b3b474e9f464633554c2f91; no voters:
I20260807 07:06:56.707044 19865 leader_election.cc:288] T 00000000000000000000000000000000 P 8416e2af8b3b474e9f464633554c2f91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 07:06:56.707193 19881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8416e2af8b3b474e9f464633554c2f91 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:06:56.707857 19881 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8416e2af8b3b474e9f464633554c2f91 [term 1 LEADER]: Becoming Leader. State: Replica: 8416e2af8b3b474e9f464633554c2f91, State: Running, Role: LEADER
I20260807 07:06:56.708458 19881 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8416e2af8b3b474e9f464633554c2f91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8416e2af8b3b474e9f464633554c2f91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42719 } }
I20260807 07:06:56.709136 19865 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8416e2af8b3b474e9f464633554c2f91 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:06:56.711669 19882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8416e2af8b3b474e9f464633554c2f91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8416e2af8b3b474e9f464633554c2f91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8416e2af8b3b474e9f464633554c2f91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42719 } } }
I20260807 07:06:56.712078 19882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8416e2af8b3b474e9f464633554c2f91 [sys.catalog]: This master's current role is: LEADER
I20260807 07:06:56.712610 19883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8416e2af8b3b474e9f464633554c2f91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8416e2af8b3b474e9f464633554c2f91. Latest consensus state: current_term: 1 leader_uuid: "8416e2af8b3b474e9f464633554c2f91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8416e2af8b3b474e9f464633554c2f91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42719 } } }
I20260807 07:06:56.712965 19883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8416e2af8b3b474e9f464633554c2f91 [sys.catalog]: This master's current role is: LEADER
I20260807 07:06:56.719678 19904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:06:56.726341 19904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:06:56.735668 19904 catalog_manager.cc:1347] Generated new cluster ID: 5a63f12688f24c65a59229fe8938ca2b
I20260807 07:06:56.735821 19904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:06:56.746927 19904 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:06:56.748123 19904 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:06:56.757872 19904 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8416e2af8b3b474e9f464633554c2f91: Generated new TSK 0
I20260807 07:06:56.758770 19904 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 07:06:56.772329 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 19765
W20260807 07:06:56.773535 19858 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46277
W20260807 07:06:56.774243 21836 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 19765: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 07:06:56.774365 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 19765
2026-08-07T07:06:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2184 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786086078080656-21836-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 07:06:56 dist-test-slave-ncp9 krb5kdc[19920](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 07:06:56 dist-test-slave-ncp9 krb5kdc[19920](info): set up 2 sockets
Aug 07 07:06:56 dist-test-slave-ncp9 krb5kdc[19920](info): commencing operation
krb5kdc: starting...
W20260807 07:06:58.021611 21836 mini_kdc.cc:121] Time spent starting KDC: real 1.202s user 0.000s sys 0.062s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Aug 07 07:06:58 dist-test-slave-ncp9 krb5kdc[19920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T07:06:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:06:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786086078080656-21836-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786086078080656-21836-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 07:06:58.374171 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44409
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786086078080656-21836-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44409
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:06:58.714193 19941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:06:58.714471 19941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:06:58.714602 19941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:06:58.721647 19941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:06:58.721766 19941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:06:58.721841 19941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:06:58.721925 19941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:06:58.721982 19941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:06:58.732455 19941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786086078080656-21836-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:06:58.733875 19941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:06:58.735162 19941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:06:58.746767 19951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:58.746734 19953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:06:58.746821 19956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:06:58.747597 19941 server_base.cc:1034] running on GCE node
I20260807 07:06:58.748270 19941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:06:58.749562 19941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:06:58.750780 19941 hybrid_clock.cc:648] HybridClock initialized: now 1786086418750737 us; error 26 us; skew 500 ppm
I20260807 07:06:58.751111 19941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:06:58 dist-test-slave-ncp9 krb5kdc[19920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:06:58.754053 19941 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 07:06:58.755384 19941 webserver.cc:460] Webserver started at http://127.0.0.1:33411/ using document root <none> and password file <none>
I20260807 07:06:58.755990 19941 fs_manager.cc:359] Metadata directory not provided
I20260807 07:06:58.756108 19941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:06:58.756378 19941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:06:58.759202 19941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "21ffb0c6d27d43179512b353fa1c900b"
format_stamp: "Formatted at 2026-08-07 07:06:58 on dist-test-slave-ncp9"
server_key: "087e5599f7508ea20380910f103ced12"
server_key_iv: "77e4c27f7fb613c9113374fa84619c4d"
server_key_version: "encryptionkey@0"
I20260807 07:06:58.759915 19941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "21ffb0c6d27d43179512b353fa1c900b"
format_stamp: "Formatted at 2026-08-07 07:06:58 on dist-test-slave-ncp9"
server_key: "087e5599f7508ea20380910f103ced12"
server_key_iv: "77e4c27f7fb613c9113374fa84619c4d"
server_key_version: "encryptionkey@0"
I20260807 07:06:58.764650 19941 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260807 07:06:58.767853 19969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:06:58.769084 19941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 07:06:58.769239 19941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "21ffb0c6d27d43179512b353fa1c900b"
format_stamp: "Formatted at 2026-08-07 07:06:58 on dist-test-slave-ncp9"
server_key: "087e5599f7508ea20380910f103ced12"
server_key_iv: "77e4c27f7fb613c9113374fa84619c4d"
server_key_version: "encryptionkey@0"
I20260807 07:06:58.769409 19941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:06:58.774973 19941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:06:58.778980 19941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:06:58.779249 19941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:06:58.797255 19941 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44409
I20260807 07:06:58.797297 20037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44409 every 8 connection(s)
I20260807 07:06:58.798557 19941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:06:58.802122 20039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:06:58.807364 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 19941
I20260807 07:06:58.807526 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:06:58.807951 21836 external_mini_cluster.cc:1428] Setting key 22547fb3dd7aa48829aabb253a16c738
I20260807 07:06:58.812855 20039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21ffb0c6d27d43179512b353fa1c900b: Bootstrap starting.
I20260807 07:06:58.815277 20046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:58.809652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34871 (local address 127.0.0.1:44409)
0807 07:06:58.810441 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:58.810452 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:58.810498 (+ 46us) server_negotiation.cc:407] Connection header received
0807 07:06:58.812224 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:58.812246 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:58.812551 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:58.812873 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:58.813818 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:58.813825 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:58.813836 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:06:58.813852 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:58.813910 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:06:58.813933 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:58.814146 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:58.814295 (+ 149us) server_negotiation.cc:299] Negotiation successful
0807 07:06:58.814456 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":352,"threads_started":1}
I20260807 07:06:58.816046 20039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21ffb0c6d27d43179512b353fa1c900b: Neither blocks nor log segments found. Creating new log.
I20260807 07:06:58.817101 20039 log.cc:824] T 00000000000000000000000000000000 P 21ffb0c6d27d43179512b353fa1c900b: Log is configured to *not* fsync() on all Append() calls
I20260807 07:06:58.818660 21836 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 07:06:58.820297 20039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21ffb0c6d27d43179512b353fa1c900b: No bootstrap required, opened a new log
I20260807 07:06:58.826776 20046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:06:58.824235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36127 (local address 127.0.0.1:44409)
0807 07:06:58.824442 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0807 07:06:58.824448 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:06:58.824683 (+ 235us) server_negotiation.cc:407] Connection header received
0807 07:06:58.824879 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:06:58.824884 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:06:58.824975 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:06:58.825072 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:06:58.826095 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:06:58.826099 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:06:58.826102 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:06:58.826122 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:06:58.826169 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:06:58.826189 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:06:58.826296 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0807 07:06:58.826412 (+ 116us) server_negotiation.cc:299] Negotiation successful
0807 07:06:58.826461 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260807 07:06:58.830242 20039 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21ffb0c6d27d43179512b353fa1c900b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21ffb0c6d27d43179512b353fa1c900b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44409 } }
I20260807 07:06:58.830631 20039 raft_consensus.cc:374] T 00000000000000000000000000000000 P 21ffb0c6d27d43179512b353fa1c900b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:06:58.830725 20039 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21ffb0c6d27d43179512b353fa1c900b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21ffb0c6d27d43179512b353fa1c900b, State: Initialized, Role: FOLLOWER
I20260807 07:06:58.831389 20039 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21ffb0c6d27d43179512b353fa1c900b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21ffb0c6d27d43179512b353fa1c900b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44409 } }
I20260807 07:06:58.831636 20039 raft_consensus.cc:388] T 00000000000000000000000000000000 P 21ffb0c6d27d43179512b353fa1c900b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:06:58.831741 20039 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21ffb0c6d27d43179512b353fa1c900b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:06:58.831868 20039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21ffb0c6d27d43179512b353fa1c900b [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:06:58.834229 20039 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21ffb0c6d27d43179512b353fa1c900b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21ffb0c6d27d43179512b353fa1c900b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44409 } }
I20260807 07:06:58.834740 20039 leader_election.cc:302] T 00000000000000000000000000000000 P 21ffb0c6d27d43179512b353fa1c900b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21ffb0c6d27d43179512b353fa1c900b; no voters:
I20260807 07:06:58.835278 20039 leader_election.cc:288] T 00000000000000000000000000000000 P 21ffb0c6d27d43179512b353fa1c900b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 07:06:58.835413 20058 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21ffb0c6d27d43179512b353fa1c900b [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:06:58.836048 20058 raft_consensus.cc:686] T 00000000000000000000000000000000 P 21ffb0c6d27d43179512b353fa1c900b [term 1 LEADER]: Becoming Leader. State: Replica: 21ffb0c6d27d43179512b353fa1c900b, State: Running, Role: LEADER
I20260807 07:06:58.836592 20058 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21ffb0c6d27d43179512b353fa1c900b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21ffb0c6d27d43179512b353fa1c900b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44409 } }
I20260807 07:06:58.837275 20039 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21ffb0c6d27d43179512b353fa1c900b [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:06:58.839394 20061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21ffb0c6d27d43179512b353fa1c900b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21ffb0c6d27d43179512b353fa1c900b. Latest consensus state: current_term: 1 leader_uuid: "21ffb0c6d27d43179512b353fa1c900b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21ffb0c6d27d43179512b353fa1c900b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44409 } } }
I20260807 07:06:58.839437 20060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21ffb0c6d27d43179512b353fa1c900b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21ffb0c6d27d43179512b353fa1c900b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21ffb0c6d27d43179512b353fa1c900b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44409 } } }
I20260807 07:06:58.840191 20060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21ffb0c6d27d43179512b353fa1c900b [sys.catalog]: This master's current role is: LEADER
I20260807 07:06:58.840725 20061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21ffb0c6d27d43179512b353fa1c900b [sys.catalog]: This master's current role is: LEADER
I20260807 07:06:58.844547 20070 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:06:58.852515 20070 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:06:58.861308 20070 catalog_manager.cc:1347] Generated new cluster ID: ebf32ee5246e45d8b296749c8baaebf0
I20260807 07:06:58.861424 20070 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:06:58.874823 20070 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:06:58.875213 20070 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:06:58.887807 20070 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 21ffb0c6d27d43179512b353fa1c900b: Generated new TSK 0
I20260807 07:06:58.888618 20070 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 07:06:58.901587 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 19941
W20260807 07:06:58.902804 20032 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34871
W20260807 07:06:58.903582 21836 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 19941: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 07:06:58.903779 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 19941
2026-08-07T07:06:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2129 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786086078080656-21836-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 07:06:59 dist-test-slave-ncp9 krb5kdc[20089](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 07:06:59 dist-test-slave-ncp9 krb5kdc[20089](info): set up 2 sockets
Aug 07 07:06:59 dist-test-slave-ncp9 krb5kdc[20089](info): commencing operation
krb5kdc: starting...
W20260807 07:07:00.148989 21836 mini_kdc.cc:121] Time spent starting KDC: real 1.199s user 0.000s sys 0.058s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Aug 07 07:07:00 dist-test-slave-ncp9 krb5kdc[20089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T07:07:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:07:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786086078080656-21836-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786086078080656-21836-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 07:07:00.495919 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45075
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786086078080656-21836-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:07:00.837081 20109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:07:00.837435 20109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:07:00.837594 20109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:07:00.845453 20109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:07:00.845592 20109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:07:00.845671 20109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:07:00.845762 20109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:07:00.845819 20109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:07:00.857111 20109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786086078080656-21836-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:07:00.858455 20109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:07:00.859748 20109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:07:00.871693 20124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:07:00.871855 20121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:07:00.871810 20122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:07:00.872684 20109 server_base.cc:1034] running on GCE node
I20260807 07:07:00.873435 20109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:07:00.874788 20109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:07:00.876003 20109 hybrid_clock.cc:648] HybridClock initialized: now 1786086420875951 us; error 46 us; skew 500 ppm
I20260807 07:07:00.876350 20109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:07:00 dist-test-slave-ncp9 krb5kdc[20089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:07:00.882314 20109 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 07:07:00.883797 20109 webserver.cc:460] Webserver started at http://127.0.0.1:38765/ using document root <none> and password file <none>
I20260807 07:07:00.884514 20109 fs_manager.cc:359] Metadata directory not provided
I20260807 07:07:00.884696 20109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:07:00.885077 20109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:07:00.888336 20109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "81d979e574274253a5f9f5f0ab91fbd4"
format_stamp: "Formatted at 2026-08-07 07:07:00 on dist-test-slave-ncp9"
server_key: "78d381e0c4fbc178f2c5a78086de9aa1"
server_key_iv: "7d420dba006aff02686fb530b9faddc3"
server_key_version: "encryptionkey@0"
I20260807 07:07:00.889256 20109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "81d979e574274253a5f9f5f0ab91fbd4"
format_stamp: "Formatted at 2026-08-07 07:07:00 on dist-test-slave-ncp9"
server_key: "78d381e0c4fbc178f2c5a78086de9aa1"
server_key_iv: "7d420dba006aff02686fb530b9faddc3"
server_key_version: "encryptionkey@0"
I20260807 07:07:00.894716 20109 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 07:07:00.898548 20132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:07:00.899904 20109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 07:07:00.900076 20109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "81d979e574274253a5f9f5f0ab91fbd4"
format_stamp: "Formatted at 2026-08-07 07:07:00 on dist-test-slave-ncp9"
server_key: "78d381e0c4fbc178f2c5a78086de9aa1"
server_key_iv: "7d420dba006aff02686fb530b9faddc3"
server_key_version: "encryptionkey@0"
I20260807 07:07:00.900269 20109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:07:00.920075 20109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:07:00.924388 20109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:07:00.924674 20109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:07:00.941658 20109 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45075
I20260807 07:07:00.941715 20218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45075 every 8 connection(s)
I20260807 07:07:00.942955 20109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:07:00.946391 20219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:07:00.950066 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 20109
I20260807 07:07:00.950228 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:07:00.950641 21836 external_mini_cluster.cc:1428] Setting key 52f9abcaeed1eb52d8ef8daaacf4b08b
I20260807 07:07:00.958539 20219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81d979e574274253a5f9f5f0ab91fbd4: Bootstrap starting.
I20260807 07:07:00.962144 20219 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81d979e574274253a5f9f5f0ab91fbd4: Neither blocks nor log segments found. Creating new log.
I20260807 07:07:00.963428 20219 log.cc:824] T 00000000000000000000000000000000 P 81d979e574274253a5f9f5f0ab91fbd4: Log is configured to *not* fsync() on all Append() calls
Aug 07 07:07:00 dist-test-slave-ncp9 krb5kdc[20089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260807 07:07:00.966513 20219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81d979e574274253a5f9f5f0ab91fbd4: No bootstrap required, opened a new log
I20260807 07:07:00.974036 20222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:00.952315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:39861 (local address 127.0.0.1:45075)
0807 07:07:00.953043 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:00.953054 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:00.953096 (+ 42us) server_negotiation.cc:407] Connection header received
0807 07:07:00.954632 (+ 1536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:00.954653 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:00.955016 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:00.955307 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:00.957204 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:00.958440 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:00.960149 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:00.960789 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:00.960944 (+ 155us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:00.965857 (+ 4913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:00.965865 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:00.965879 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:07:00.965917 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:00.969119 (+ 3202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:00.970084 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:00.970095 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:00.970102 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:00.970302 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:00.970781 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:00.970785 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:00.970787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:00.971478 (+ 691us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:00.971885 (+ 407us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:00.972369 (+ 484us) server_negotiation.cc:299] Negotiation successful
0807 07:07:00.972855 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":301,"threads_started":1}
I20260807 07:07:00.975126 20219 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81d979e574274253a5f9f5f0ab91fbd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d979e574274253a5f9f5f0ab91fbd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45075 } }
I20260807 07:07:00.975610 20219 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81d979e574274253a5f9f5f0ab91fbd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:07:00.975720 20219 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81d979e574274253a5f9f5f0ab91fbd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81d979e574274253a5f9f5f0ab91fbd4, State: Initialized, Role: FOLLOWER
I20260807 07:07:00.976323 20219 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81d979e574274253a5f9f5f0ab91fbd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d979e574274253a5f9f5f0ab91fbd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45075 } }
I20260807 07:07:00.976588 20219 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81d979e574274253a5f9f5f0ab91fbd4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:07:00.976758 20219 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81d979e574274253a5f9f5f0ab91fbd4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:07:00.976974 20219 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81d979e574274253a5f9f5f0ab91fbd4 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:07:00.978224 21836 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 07:07:00.979445 20219 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81d979e574274253a5f9f5f0ab91fbd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d979e574274253a5f9f5f0ab91fbd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45075 } }
I20260807 07:07:00.979923 20219 leader_election.cc:302] T 00000000000000000000000000000000 P 81d979e574274253a5f9f5f0ab91fbd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81d979e574274253a5f9f5f0ab91fbd4; no voters:
I20260807 07:07:00.980530 20219 leader_election.cc:288] T 00000000000000000000000000000000 P 81d979e574274253a5f9f5f0ab91fbd4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 07:07:00.980650 20224 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81d979e574274253a5f9f5f0ab91fbd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:07:00.982223 20224 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81d979e574274253a5f9f5f0ab91fbd4 [term 1 LEADER]: Becoming Leader. State: Replica: 81d979e574274253a5f9f5f0ab91fbd4, State: Running, Role: LEADER
I20260807 07:07:00.982944 20224 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81d979e574274253a5f9f5f0ab91fbd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d979e574274253a5f9f5f0ab91fbd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45075 } }
I20260807 07:07:00.984608 20219 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81d979e574274253a5f9f5f0ab91fbd4 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:07:00.985795 20230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81d979e574274253a5f9f5f0ab91fbd4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81d979e574274253a5f9f5f0ab91fbd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d979e574274253a5f9f5f0ab91fbd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45075 } } }
I20260807 07:07:00.986943 20230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81d979e574274253a5f9f5f0ab91fbd4 [sys.catalog]: This master's current role is: LEADER
I20260807 07:07:00.986598 20232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81d979e574274253a5f9f5f0ab91fbd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81d979e574274253a5f9f5f0ab91fbd4. Latest consensus state: current_term: 1 leader_uuid: "81d979e574274253a5f9f5f0ab91fbd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d979e574274253a5f9f5f0ab91fbd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45075 } } }
I20260807 07:07:00.987546 20232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81d979e574274253a5f9f5f0ab91fbd4 [sys.catalog]: This master's current role is: LEADER
I20260807 07:07:00.988409 20240 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:07:00.994953 20240 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:07:01.002858 20222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:00.982599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:38275 (local address 127.0.0.1:45075)
0807 07:07:00.982780 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:00.982787 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:00.986368 (+ 3581us) server_negotiation.cc:407] Connection header received
0807 07:07:00.986511 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:00.986514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:00.986606 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:00.986716 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:00.989461 (+ 2745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:00.990230 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:00.991433 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:00.992103 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:00.992217 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:00.995412 (+ 3195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:00.995420 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:00.995425 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:07:00.995463 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:00.999142 (+ 3679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:01.000139 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:01.000145 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:01.000148 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:01.000295 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:01.000883 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:01.000910 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:01.000913 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:01.001361 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:01.001623 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:01.002288 (+ 665us) server_negotiation.cc:299] Negotiation successful
0807 07:07:01.002473 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 07:07:01.008846 20240 catalog_manager.cc:1347] Generated new cluster ID: 73b6752382a544a8b4bf1aabcf0cbd13
I20260807 07:07:01.009024 20240 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:07:01.037720 20240 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:07:01.039150 20240 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:07:01.057530 20240 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81d979e574274253a5f9f5f0ab91fbd4: Generated new TSK 0
I20260807 07:07:01.058528 20240 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 07:07:01.072396 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 20109
I20260807 07:07:01.073779 20213 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:07:01.168475 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 20109
W20260807 07:07:01.190781 20086 connection.cc:535] client connection to 127.0.0.1:45075 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45075): Connection reset by peer (error 104)
2026-08-07T07:07:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2260 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786086078080656-21836-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 07:07:01 dist-test-slave-ncp9 krb5kdc[20262](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 07:07:01 dist-test-slave-ncp9 krb5kdc[20262](info): set up 2 sockets
Aug 07 07:07:01 dist-test-slave-ncp9 krb5kdc[20262](info): commencing operation
krb5kdc: starting...
W20260807 07:07:02.398828 21836 mini_kdc.cc:121] Time spent starting KDC: real 1.189s user 0.001s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Aug 07 07:07:02 dist-test-slave-ncp9 krb5kdc[20262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T07:07:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:07:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786086078080656-21836-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786086078080656-21836-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 07:07:02.752009 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42417
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786086078080656-21836-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42417
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:07:03.091351 20278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:07:03.091666 20278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:07:03.091776 20278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:07:03.099108 20278 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:07:03.099215 20278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:07:03.099284 20278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:07:03.099346 20278 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:07:03.099390 20278 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:07:03.109586 20278 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42417
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786086078080656-21836-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:07:03.110862 20278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:07:03.112099 20278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:07:03.124214 20283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:07:03.124321 20289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:07:03.124356 20285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:07:03.125370 20278 server_base.cc:1034] running on GCE node
I20260807 07:07:03.126151 20278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:07:03.127584 20278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:07:03.128789 20278 hybrid_clock.cc:648] HybridClock initialized: now 1786086423128737 us; error 45 us; skew 500 ppm
I20260807 07:07:03.129145 20278 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:07:03 dist-test-slave-ncp9 krb5kdc[20262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086423, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:07:03.132257 20278 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 07:07:03.133841 20278 webserver.cc:460] Webserver started at http://127.0.0.1:33191/ using document root <none> and password file <none>
I20260807 07:07:03.134485 20278 fs_manager.cc:359] Metadata directory not provided
I20260807 07:07:03.134600 20278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:07:03.134887 20278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:07:03.137854 20278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "453c1a8819f242cb9f381c357d9b0e2c"
format_stamp: "Formatted at 2026-08-07 07:07:03 on dist-test-slave-ncp9"
server_key: "970af245ec5f77f0df14967b74122596"
server_key_iv: "fcf84cb61a6bf7049834ddcd4250e056"
server_key_version: "encryptionkey@0"
I20260807 07:07:03.138597 20278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "453c1a8819f242cb9f381c357d9b0e2c"
format_stamp: "Formatted at 2026-08-07 07:07:03 on dist-test-slave-ncp9"
server_key: "970af245ec5f77f0df14967b74122596"
server_key_iv: "fcf84cb61a6bf7049834ddcd4250e056"
server_key_version: "encryptionkey@0"
I20260807 07:07:03.143781 20278 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 07:07:03.147643 20299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:07:03.149188 20278 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260807 07:07:03.149394 20278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "453c1a8819f242cb9f381c357d9b0e2c"
format_stamp: "Formatted at 2026-08-07 07:07:03 on dist-test-slave-ncp9"
server_key: "970af245ec5f77f0df14967b74122596"
server_key_iv: "fcf84cb61a6bf7049834ddcd4250e056"
server_key_version: "encryptionkey@0"
I20260807 07:07:03.149616 20278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:07:03.167897 20278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:07:03.172502 20278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:07:03.172806 20278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:07:03.191133 20278 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42417
I20260807 07:07:03.191165 20378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42417 every 8 connection(s)
I20260807 07:07:03.192435 20278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:07:03.194486 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 20278
I20260807 07:07:03.194654 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:07:03.195065 21836 external_mini_cluster.cc:1428] Setting key bd20d86fc6755ddaf53ebc515e380fbc
I20260807 07:07:03.196404 20379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:07:03.209597 20382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:03.196809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:41319 (local address 127.0.0.1:42417)
0807 07:07:03.197615 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:03.197630 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:03.197682 (+ 52us) server_negotiation.cc:407] Connection header received
0807 07:07:03.199633 (+ 1951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:03.199661 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:03.200184 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:03.200622 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:03.201179 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:03.203201 (+ 2022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:03.204472 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:03.205508 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:03.205687 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:03.206788 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:03.206798 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:03.206816 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:07:03.207553 (+ 737us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:03.207653 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:07:03.207687 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:03.207858 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:03.208115 (+ 257us) server_negotiation.cc:299] Negotiation successful
0807 07:07:03.208294 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":305,"threads_started":1}
I20260807 07:07:03.211051 20379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 453c1a8819f242cb9f381c357d9b0e2c: Bootstrap starting.
I20260807 07:07:03.213450 21836 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 07:07:03.214605 20379 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 453c1a8819f242cb9f381c357d9b0e2c: Neither blocks nor log segments found. Creating new log.
I20260807 07:07:03.216135 20379 log.cc:824] T 00000000000000000000000000000000 P 453c1a8819f242cb9f381c357d9b0e2c: Log is configured to *not* fsync() on all Append() calls
I20260807 07:07:03.220988 20379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 453c1a8819f242cb9f381c357d9b0e2c: No bootstrap required, opened a new log
I20260807 07:07:03.227263 20382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:03.220154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:58069 (local address 127.0.0.1:42417)
0807 07:07:03.220347 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:03.220353 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:03.220681 (+ 328us) server_negotiation.cc:407] Connection header received
0807 07:07:03.220857 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:03.220862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:03.220974 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:03.221103 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:03.221648 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:03.222776 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:03.224205 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:03.224971 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:03.225116 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:03.226448 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:03.226454 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:03.226459 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:07:03.226489 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:03.226550 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:07:03.226571 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:03.226687 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:03.226823 (+ 136us) server_negotiation.cc:299] Negotiation successful
0807 07:07:03.226885 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 07:07:03.230922 20379 raft_consensus.cc:348] T 00000000000000000000000000000000 P 453c1a8819f242cb9f381c357d9b0e2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "453c1a8819f242cb9f381c357d9b0e2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42417 } }
I20260807 07:07:03.231359 20379 raft_consensus.cc:374] T 00000000000000000000000000000000 P 453c1a8819f242cb9f381c357d9b0e2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:07:03.231437 20379 raft_consensus.cc:729] T 00000000000000000000000000000000 P 453c1a8819f242cb9f381c357d9b0e2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 453c1a8819f242cb9f381c357d9b0e2c, State: Initialized, Role: FOLLOWER
I20260807 07:07:03.231999 20379 consensus_queue.cc:260] T 00000000000000000000000000000000 P 453c1a8819f242cb9f381c357d9b0e2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "453c1a8819f242cb9f381c357d9b0e2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42417 } }
I20260807 07:07:03.232223 20379 raft_consensus.cc:388] T 00000000000000000000000000000000 P 453c1a8819f242cb9f381c357d9b0e2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:07:03.232327 20379 raft_consensus.cc:482] T 00000000000000000000000000000000 P 453c1a8819f242cb9f381c357d9b0e2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:07:03.232486 20379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 453c1a8819f242cb9f381c357d9b0e2c [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:07:03.235773 20379 raft_consensus.cc:504] T 00000000000000000000000000000000 P 453c1a8819f242cb9f381c357d9b0e2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "453c1a8819f242cb9f381c357d9b0e2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42417 } }
I20260807 07:07:03.236311 20379 leader_election.cc:302] T 00000000000000000000000000000000 P 453c1a8819f242cb9f381c357d9b0e2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 453c1a8819f242cb9f381c357d9b0e2c; no voters:
I20260807 07:07:03.236884 20379 leader_election.cc:288] T 00000000000000000000000000000000 P 453c1a8819f242cb9f381c357d9b0e2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 07:07:03.237067 20395 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 453c1a8819f242cb9f381c357d9b0e2c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:07:03.237757 20395 raft_consensus.cc:686] T 00000000000000000000000000000000 P 453c1a8819f242cb9f381c357d9b0e2c [term 1 LEADER]: Becoming Leader. State: Replica: 453c1a8819f242cb9f381c357d9b0e2c, State: Running, Role: LEADER
I20260807 07:07:03.238329 20395 consensus_queue.cc:237] T 00000000000000000000000000000000 P 453c1a8819f242cb9f381c357d9b0e2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "453c1a8819f242cb9f381c357d9b0e2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42417 } }
I20260807 07:07:03.239575 20379 sys_catalog.cc:564] T 00000000000000000000000000000000 P 453c1a8819f242cb9f381c357d9b0e2c [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:07:03.241462 20397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 453c1a8819f242cb9f381c357d9b0e2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 453c1a8819f242cb9f381c357d9b0e2c. Latest consensus state: current_term: 1 leader_uuid: "453c1a8819f242cb9f381c357d9b0e2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "453c1a8819f242cb9f381c357d9b0e2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42417 } } }
I20260807 07:07:03.241428 20396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 453c1a8819f242cb9f381c357d9b0e2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "453c1a8819f242cb9f381c357d9b0e2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "453c1a8819f242cb9f381c357d9b0e2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42417 } } }
I20260807 07:07:03.241936 20397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 453c1a8819f242cb9f381c357d9b0e2c [sys.catalog]: This master's current role is: LEADER
I20260807 07:07:03.241943 20396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 453c1a8819f242cb9f381c357d9b0e2c [sys.catalog]: This master's current role is: LEADER
I20260807 07:07:03.243338 20403 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:07:03.251754 20403 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:07:03.263486 20403 catalog_manager.cc:1347] Generated new cluster ID: bf2ddcfecfb54ac8b66209c3d764b000
I20260807 07:07:03.263628 20403 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:07:03.279095 20403 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:07:03.281047 20403 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:07:03.290306 20403 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 453c1a8819f242cb9f381c357d9b0e2c: Generated new TSK 0
I20260807 07:07:03.291493 20403 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 07:07:03.308171 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 20278
W20260807 07:07:03.309762 20373 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.213:41319
W20260807 07:07:03.310585 21836 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 20278: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 07:07:03.310750 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 20278
W20260807 07:07:03.335045 20258 connection.cc:535] client connection to 127.0.0.1:42417 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42417): Connection reset by peer (error 104)
2026-08-07T07:07:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2144 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786086078080656-21836-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 07:07:03 dist-test-slave-ncp9 krb5kdc[20424](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 07:07:03 dist-test-slave-ncp9 krb5kdc[20424](info): set up 2 sockets
Aug 07 07:07:03 dist-test-slave-ncp9 krb5kdc[20424](info): commencing operation
krb5kdc: starting...
W20260807 07:07:04.535125 21836 mini_kdc.cc:121] Time spent starting KDC: real 1.181s user 0.001s sys 0.057s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Aug 07 07:07:04 dist-test-slave-ncp9 krb5kdc[20424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T07:07:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:07:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786086078080656-21836-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786086078080656-21836-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 07:07:04.901757 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44773
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786086078080656-21836-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44773
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:07:05.247165 20444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:07:05.247502 20444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:07:05.247629 20444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:07:05.255261 20444 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:07:05.255420 20444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:07:05.255499 20444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:07:05.255580 20444 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:07:05.255640 20444 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:07:05.266290 20444 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786086078080656-21836-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:07:05.267709 20444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:07:05.269132 20444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:07:05.281121 20450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:07:05.281162 20453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:07:05.281199 20451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:07:05.282229 20444 server_base.cc:1034] running on GCE node
I20260807 07:07:05.283079 20444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:07:05.284623 20444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:07:05.285823 20444 hybrid_clock.cc:648] HybridClock initialized: now 1786086425285757 us; error 66 us; skew 500 ppm
I20260807 07:07:05.286191 20444 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:07:05 dist-test-slave-ncp9 krb5kdc[20424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086425, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:07:05.289439 20444 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 07:07:05.291448 20444 webserver.cc:460] Webserver started at http://127.0.0.1:33807/ using document root <none> and password file <none>
I20260807 07:07:05.292129 20444 fs_manager.cc:359] Metadata directory not provided
I20260807 07:07:05.292248 20444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:07:05.292510 20444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:07:05.295719 20444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "451f5892aa1e4f479acb279dd98f61ce"
format_stamp: "Formatted at 2026-08-07 07:07:05 on dist-test-slave-ncp9"
server_key: "c71cd1d19e0fdaa11eb27d9b0e3ca7d4"
server_key_iv: "f6bca98e0a1addf57bece43ea4fd781d"
server_key_version: "encryptionkey@0"
I20260807 07:07:05.296504 20444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "451f5892aa1e4f479acb279dd98f61ce"
format_stamp: "Formatted at 2026-08-07 07:07:05 on dist-test-slave-ncp9"
server_key: "c71cd1d19e0fdaa11eb27d9b0e3ca7d4"
server_key_iv: "f6bca98e0a1addf57bece43ea4fd781d"
server_key_version: "encryptionkey@0"
I20260807 07:07:05.301980 20444 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260807 07:07:05.305624 20463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:07:05.306983 20444 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 07:07:05.307165 20444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "451f5892aa1e4f479acb279dd98f61ce"
format_stamp: "Formatted at 2026-08-07 07:07:05 on dist-test-slave-ncp9"
server_key: "c71cd1d19e0fdaa11eb27d9b0e3ca7d4"
server_key_iv: "f6bca98e0a1addf57bece43ea4fd781d"
server_key_version: "encryptionkey@0"
I20260807 07:07:05.307365 20444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:07:05.313412 20444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:07:05.317404 20444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:07:05.317677 20444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:07:05.335883 20444 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44773
I20260807 07:07:05.335922 20545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44773 every 8 connection(s)
I20260807 07:07:05.337338 20444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:07:05.341171 20546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:07:05.347748 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 20444
I20260807 07:07:05.347965 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:07:05.348512 21836 external_mini_cluster.cc:1428] Setting key ed36fbfbb425f08b349857b124168dfe
I20260807 07:07:05.352932 20546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 451f5892aa1e4f479acb279dd98f61ce: Bootstrap starting.
I20260807 07:07:05.356729 20546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 451f5892aa1e4f479acb279dd98f61ce: Neither blocks nor log segments found. Creating new log.
I20260807 07:07:05.356945 20551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:05.350612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:50793 (local address 127.0.0.1:44773)
0807 07:07:05.351504 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:05.351517 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:05.351563 (+ 46us) server_negotiation.cc:407] Connection header received
0807 07:07:05.353431 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:05.353458 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:05.353842 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:05.354184 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:05.355325 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:05.355331 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:05.355344 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:07:05.355359 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:05.355422 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:07:05.355442 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:05.355637 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:05.355742 (+ 105us) server_negotiation.cc:299] Negotiation successful
0807 07:07:05.355929 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":393,"threads_started":1}
I20260807 07:07:05.358178 20546 log.cc:824] T 00000000000000000000000000000000 P 451f5892aa1e4f479acb279dd98f61ce: Log is configured to *not* fsync() on all Append() calls
I20260807 07:07:05.361117 21836 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 07:07:05.361565 20546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 451f5892aa1e4f479acb279dd98f61ce: No bootstrap required, opened a new log
I20260807 07:07:05.368847 20551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:05.365942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:44307 (local address 127.0.0.1:44773)
0807 07:07:05.366164 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:05.366170 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:05.366678 (+ 508us) server_negotiation.cc:407] Connection header received
0807 07:07:05.366840 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:05.366844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:05.366906 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:05.367007 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:05.368031 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:05.368036 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:05.368039 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:07:05.368060 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:05.368108 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:07:05.368124 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:05.368233 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:05.368441 (+ 208us) server_negotiation.cc:299] Negotiation successful
0807 07:07:05.368515 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260807 07:07:05.371232 20546 raft_consensus.cc:348] T 00000000000000000000000000000000 P 451f5892aa1e4f479acb279dd98f61ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451f5892aa1e4f479acb279dd98f61ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44773 } }
I20260807 07:07:05.371685 20546 raft_consensus.cc:374] T 00000000000000000000000000000000 P 451f5892aa1e4f479acb279dd98f61ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:07:05.371786 20546 raft_consensus.cc:729] T 00000000000000000000000000000000 P 451f5892aa1e4f479acb279dd98f61ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 451f5892aa1e4f479acb279dd98f61ce, State: Initialized, Role: FOLLOWER
I20260807 07:07:05.372432 20546 consensus_queue.cc:260] T 00000000000000000000000000000000 P 451f5892aa1e4f479acb279dd98f61ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451f5892aa1e4f479acb279dd98f61ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44773 } }
I20260807 07:07:05.372699 20546 raft_consensus.cc:388] T 00000000000000000000000000000000 P 451f5892aa1e4f479acb279dd98f61ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:07:05.372836 20546 raft_consensus.cc:482] T 00000000000000000000000000000000 P 451f5892aa1e4f479acb279dd98f61ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:07:05.373032 20546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 451f5892aa1e4f479acb279dd98f61ce [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:07:05.375617 20546 raft_consensus.cc:504] T 00000000000000000000000000000000 P 451f5892aa1e4f479acb279dd98f61ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451f5892aa1e4f479acb279dd98f61ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44773 } }
I20260807 07:07:05.376279 20546 leader_election.cc:302] T 00000000000000000000000000000000 P 451f5892aa1e4f479acb279dd98f61ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 451f5892aa1e4f479acb279dd98f61ce; no voters:
I20260807 07:07:05.377054 20546 leader_election.cc:288] T 00000000000000000000000000000000 P 451f5892aa1e4f479acb279dd98f61ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 07:07:05.377197 20564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 451f5892aa1e4f479acb279dd98f61ce [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:07:05.377852 20564 raft_consensus.cc:686] T 00000000000000000000000000000000 P 451f5892aa1e4f479acb279dd98f61ce [term 1 LEADER]: Becoming Leader. State: Replica: 451f5892aa1e4f479acb279dd98f61ce, State: Running, Role: LEADER
I20260807 07:07:05.378410 20564 consensus_queue.cc:237] T 00000000000000000000000000000000 P 451f5892aa1e4f479acb279dd98f61ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451f5892aa1e4f479acb279dd98f61ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44773 } }
I20260807 07:07:05.379127 20546 sys_catalog.cc:564] T 00000000000000000000000000000000 P 451f5892aa1e4f479acb279dd98f61ce [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:07:05.381708 20570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 451f5892aa1e4f479acb279dd98f61ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 451f5892aa1e4f479acb279dd98f61ce. Latest consensus state: current_term: 1 leader_uuid: "451f5892aa1e4f479acb279dd98f61ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451f5892aa1e4f479acb279dd98f61ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44773 } } }
I20260807 07:07:05.382329 20570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 451f5892aa1e4f479acb279dd98f61ce [sys.catalog]: This master's current role is: LEADER
I20260807 07:07:05.382856 20565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 451f5892aa1e4f479acb279dd98f61ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "451f5892aa1e4f479acb279dd98f61ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451f5892aa1e4f479acb279dd98f61ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44773 } } }
I20260807 07:07:05.383246 20565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 451f5892aa1e4f479acb279dd98f61ce [sys.catalog]: This master's current role is: LEADER
I20260807 07:07:05.384109 20580 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:07:05.389832 20580 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:07:05.399930 20580 catalog_manager.cc:1347] Generated new cluster ID: 433d8411702040d18467d1e0f2f00ed0
I20260807 07:07:05.400048 20580 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:07:05.435141 20580 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:07:05.435626 20580 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:07:05.450448 20580 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 451f5892aa1e4f479acb279dd98f61ce: Generated new TSK 0
I20260807 07:07:05.451565 20580 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 07:07:05.462510 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 20444
W20260807 07:07:05.463697 20539 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.213:50793
W20260807 07:07:05.464445 21836 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 20444: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 07:07:05.464596 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 20444
2026-08-07T07:07:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2156 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786086078080656-21836-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 07:07:05 dist-test-slave-ncp9 krb5kdc[20598](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 07:07:05 dist-test-slave-ncp9 krb5kdc[20598](info): set up 2 sockets
Aug 07 07:07:05 dist-test-slave-ncp9 krb5kdc[20598](info): commencing operation
krb5kdc: starting...
W20260807 07:07:06.706187 21836 mini_kdc.cc:121] Time spent starting KDC: real 1.196s user 0.004s sys 0.056s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Aug 07 07:07:06 dist-test-slave-ncp9 krb5kdc[20598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T07:07:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:07:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786086078080656-21836-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786086078080656-21836-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 07:07:07.069979 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38081
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786086078080656-21836-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38081
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:07:07.426370 20619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:07:07.426702 20619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:07:07.426837 20619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:07:07.434545 20619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:07:07.434696 20619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:07:07.434784 20619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:07:07.434867 20619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:07:07.434916 20619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:07:07.446885 20619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38081
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786086078080656-21836-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:07:07.448531 20619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:07:07.450001 20619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:07:07.462250 20626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:07:07.462224 20627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:07:07.462463 20629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:07:07.463168 20619 server_base.cc:1034] running on GCE node
I20260807 07:07:07.463966 20619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:07:07.465442 20619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:07:07.466645 20619 hybrid_clock.cc:648] HybridClock initialized: now 1786086427466602 us; error 33 us; skew 500 ppm
I20260807 07:07:07.466985 20619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:07:07 dist-test-slave-ncp9 krb5kdc[20598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086427, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:07:07.469978 20619 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 07:07:07.471350 20619 webserver.cc:460] Webserver started at http://127.0.0.1:43125/ using document root <none> and password file <none>
I20260807 07:07:07.471985 20619 fs_manager.cc:359] Metadata directory not provided
I20260807 07:07:07.472110 20619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:07:07.472406 20619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:07:07.475320 20619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "78f91908dd7b490b9feda8c2195105bb"
format_stamp: "Formatted at 2026-08-07 07:07:07 on dist-test-slave-ncp9"
server_key: "4d4826f39e66c0016930101b09186370"
server_key_iv: "162a0b303261c8a0e188e15c4bf57959"
server_key_version: "encryptionkey@0"
I20260807 07:07:07.476104 20619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "78f91908dd7b490b9feda8c2195105bb"
format_stamp: "Formatted at 2026-08-07 07:07:07 on dist-test-slave-ncp9"
server_key: "4d4826f39e66c0016930101b09186370"
server_key_iv: "162a0b303261c8a0e188e15c4bf57959"
server_key_version: "encryptionkey@0"
I20260807 07:07:07.480912 20619 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260807 07:07:07.484160 20637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:07:07.485386 20619 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 07:07:07.485546 20619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "78f91908dd7b490b9feda8c2195105bb"
format_stamp: "Formatted at 2026-08-07 07:07:07 on dist-test-slave-ncp9"
server_key: "4d4826f39e66c0016930101b09186370"
server_key_iv: "162a0b303261c8a0e188e15c4bf57959"
server_key_version: "encryptionkey@0"
I20260807 07:07:07.485721 20619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:07:07.505168 20619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:07:07.509197 20619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:07:07.509476 20619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:07:07.528247 20619 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38081
I20260807 07:07:07.528295 20717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38081 every 8 connection(s)
I20260807 07:07:07.529647 20619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:07:07.533279 20718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:07:07.537262 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 20619
I20260807 07:07:07.537443 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:07:07.537874 21836 external_mini_cluster.cc:1428] Setting key 67620cd9b44cea2b431a3a312332495a
I20260807 07:07:07.546746 20718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78f91908dd7b490b9feda8c2195105bb: Bootstrap starting.
I20260807 07:07:07.550278 20718 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78f91908dd7b490b9feda8c2195105bb: Neither blocks nor log segments found. Creating new log.
I20260807 07:07:07.551227 20718 log.cc:824] T 00000000000000000000000000000000 P 78f91908dd7b490b9feda8c2195105bb: Log is configured to *not* fsync() on all Append() calls
Aug 07 07:07:07 dist-test-slave-ncp9 krb5kdc[20598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260807 07:07:07.554220 20718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78f91908dd7b490b9feda8c2195105bb: No bootstrap required, opened a new log
I20260807 07:07:07.561875 20722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:07.539836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:43987 (local address 127.0.0.1:38081)
0807 07:07:07.540654 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:07.540669 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:07.540723 (+ 54us) server_negotiation.cc:407] Connection header received
0807 07:07:07.542553 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:07.542584 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:07.543156 (+ 572us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:07.543587 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:07.544969 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:07.546874 (+ 1905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:07.548094 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:07.549121 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:07.549322 (+ 201us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:07.553658 (+ 4336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:07.553666 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:07.553679 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:07:07.553713 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:07.557386 (+ 3673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:07.558090 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:07.558099 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:07.558106 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:07.558328 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:07.558718 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:07.558723 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:07.558727 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:07.559503 (+ 776us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:07.559832 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:07.560188 (+ 356us) server_negotiation.cc:299] Negotiation successful
0807 07:07:07.560541 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":340,"threads_started":1}
I20260807 07:07:07.563966 20718 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78f91908dd7b490b9feda8c2195105bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78f91908dd7b490b9feda8c2195105bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38081 } }
I20260807 07:07:07.564318 20718 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78f91908dd7b490b9feda8c2195105bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:07:07.564416 20718 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78f91908dd7b490b9feda8c2195105bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78f91908dd7b490b9feda8c2195105bb, State: Initialized, Role: FOLLOWER
I20260807 07:07:07.565021 20718 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78f91908dd7b490b9feda8c2195105bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78f91908dd7b490b9feda8c2195105bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38081 } }
I20260807 07:07:07.565246 20718 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78f91908dd7b490b9feda8c2195105bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:07:07.565376 20718 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78f91908dd7b490b9feda8c2195105bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:07:07.565531 20718 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78f91908dd7b490b9feda8c2195105bb [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:07:07.565598 21836 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 07:07:07.568562 20718 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78f91908dd7b490b9feda8c2195105bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78f91908dd7b490b9feda8c2195105bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38081 } }
I20260807 07:07:07.569171 20718 leader_election.cc:302] T 00000000000000000000000000000000 P 78f91908dd7b490b9feda8c2195105bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78f91908dd7b490b9feda8c2195105bb; no voters:
I20260807 07:07:07.569829 20718 leader_election.cc:288] T 00000000000000000000000000000000 P 78f91908dd7b490b9feda8c2195105bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 07:07:07.569895 20729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78f91908dd7b490b9feda8c2195105bb [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:07:07.572077 20718 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78f91908dd7b490b9feda8c2195105bb [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:07:07.572186 20729 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78f91908dd7b490b9feda8c2195105bb [term 1 LEADER]: Becoming Leader. State: Replica: 78f91908dd7b490b9feda8c2195105bb, State: Running, Role: LEADER
I20260807 07:07:07.572831 20729 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78f91908dd7b490b9feda8c2195105bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78f91908dd7b490b9feda8c2195105bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38081 } }
I20260807 07:07:07.575616 20731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78f91908dd7b490b9feda8c2195105bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78f91908dd7b490b9feda8c2195105bb. Latest consensus state: current_term: 1 leader_uuid: "78f91908dd7b490b9feda8c2195105bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78f91908dd7b490b9feda8c2195105bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38081 } } }
I20260807 07:07:07.576006 20731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78f91908dd7b490b9feda8c2195105bb [sys.catalog]: This master's current role is: LEADER
I20260807 07:07:07.575598 20730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78f91908dd7b490b9feda8c2195105bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78f91908dd7b490b9feda8c2195105bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78f91908dd7b490b9feda8c2195105bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38081 } } }
I20260807 07:07:07.576638 20730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78f91908dd7b490b9feda8c2195105bb [sys.catalog]: This master's current role is: LEADER
I20260807 07:07:07.582129 20739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:07:07.587863 20739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:07:07.590277 20722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:07.571191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:49409 (local address 127.0.0.1:38081)
0807 07:07:07.571568 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:07.571573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:07.571593 (+ 20us) server_negotiation.cc:407] Connection header received
0807 07:07:07.571646 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:07.571651 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:07.571742 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:07.571845 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:07.573737 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:07.575032 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:07.576979 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:07.577714 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:07.577839 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:07.581387 (+ 3548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:07.581395 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:07.581404 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:07:07.581446 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:07.585375 (+ 3929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:07.586392 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:07.586402 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:07.586406 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:07.586568 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:07.587129 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:07.587134 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:07.587138 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:07.587592 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:07.587889 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:07.589586 (+ 1697us) server_negotiation.cc:299] Negotiation successful
0807 07:07:07.589804 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261}
I20260807 07:07:07.597540 20739 catalog_manager.cc:1347] Generated new cluster ID: e4f467cca8b0461e87e98d6e08df74fb
I20260807 07:07:07.597659 20739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:07:07.657527 20739 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:07:07.658892 20739 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:07:07.670600 20739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78f91908dd7b490b9feda8c2195105bb: Generated new TSK 0
I20260807 07:07:07.671527 20739 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 07:07:07.684135 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 20619
I20260807 07:07:07.685292 20712 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:07:07.782316 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 20619
W20260807 07:07:07.805299 20595 connection.cc:535] client connection to 127.0.0.1:38081 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38081): Connection reset by peer (error 104)
2026-08-07T07:07:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2318 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786086078080656-21836-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 07:07:07 dist-test-slave-ncp9 krb5kdc[20762](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 07:07:07 dist-test-slave-ncp9 krb5kdc[20762](info): set up 2 sockets
Aug 07 07:07:07 dist-test-slave-ncp9 krb5kdc[20762](info): commencing operation
krb5kdc: starting...
W20260807 07:07:09.013940 21836 mini_kdc.cc:121] Time spent starting KDC: real 1.185s user 0.003s sys 0.058s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Aug 07 07:07:09 dist-test-slave-ncp9 krb5kdc[20762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T07:07:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:07:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786086078080656-21836-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786086078080656-21836-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 07:07:09.325052 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42845
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786086078080656-21836-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:07:09.663674 20778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:07:09.663926 20778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:07:09.664023 20778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:07:09.671006 20778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:07:09.671101 20778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:07:09.671159 20778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:07:09.671231 20778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:07:09.671273 20778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:07:09.680505 20778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42845
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786086078080656-21836-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:07:09.681581 20778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:07:09.682751 20778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:07:09.693334 20786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:07:09.693365 20787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:07:09.693395 20789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:07:09.694170 20778 server_base.cc:1034] running on GCE node
I20260807 07:07:09.694845 20778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:07:09.696020 20778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:07:09.697208 20778 hybrid_clock.cc:648] HybridClock initialized: now 1786086429697164 us; error 47 us; skew 500 ppm
I20260807 07:07:09.697504 20778 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:07:09 dist-test-slave-ncp9 krb5kdc[20762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086429, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:07:09.700321 20778 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 07:07:09.701619 20778 webserver.cc:460] Webserver started at http://127.0.0.1:43721/ using document root <none> and password file <none>
I20260807 07:07:09.702219 20778 fs_manager.cc:359] Metadata directory not provided
I20260807 07:07:09.702317 20778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:07:09.702548 20778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:07:09.705293 20778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "55a22737589341fe839205c3f3bd8d86"
format_stamp: "Formatted at 2026-08-07 07:07:09 on dist-test-slave-ncp9"
server_key: "4c7be96bd73cf37cb2b9decc44900e17"
server_key_iv: "49deb50dec28028813abfe021e66bdb2"
server_key_version: "encryptionkey@0"
I20260807 07:07:09.705955 20778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "55a22737589341fe839205c3f3bd8d86"
format_stamp: "Formatted at 2026-08-07 07:07:09 on dist-test-slave-ncp9"
server_key: "4c7be96bd73cf37cb2b9decc44900e17"
server_key_iv: "49deb50dec28028813abfe021e66bdb2"
server_key_version: "encryptionkey@0"
I20260807 07:07:09.710505 20778 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 07:07:09.713583 20803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:07:09.714735 20778 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 07:07:09.714874 20778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "55a22737589341fe839205c3f3bd8d86"
format_stamp: "Formatted at 2026-08-07 07:07:09 on dist-test-slave-ncp9"
server_key: "4c7be96bd73cf37cb2b9decc44900e17"
server_key_iv: "49deb50dec28028813abfe021e66bdb2"
server_key_version: "encryptionkey@0"
I20260807 07:07:09.715044 20778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:07:09.729028 20778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:07:09.733569 20778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:07:09.733861 20778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:07:09.751859 20778 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42845
I20260807 07:07:09.751904 20882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42845 every 8 connection(s)
I20260807 07:07:09.753204 20778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:07:09.756677 20885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:07:09.763528 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 20778
I20260807 07:07:09.763724 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:07:09.764254 21836 external_mini_cluster.cc:1428] Setting key 6651c341fd16d9569893f4e66eba243d
I20260807 07:07:09.767083 20885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55a22737589341fe839205c3f3bd8d86: Bootstrap starting.
I20260807 07:07:09.770241 20885 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55a22737589341fe839205c3f3bd8d86: Neither blocks nor log segments found. Creating new log.
I20260807 07:07:09.771332 20885 log.cc:824] T 00000000000000000000000000000000 P 55a22737589341fe839205c3f3bd8d86: Log is configured to *not* fsync() on all Append() calls
I20260807 07:07:09.774253 20885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55a22737589341fe839205c3f3bd8d86: No bootstrap required, opened a new log
I20260807 07:07:09.776765 20888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:09.765921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:37071 (local address 127.0.0.1:42845)
0807 07:07:09.766627 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:09.766638 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:09.766684 (+ 46us) server_negotiation.cc:407] Connection header received
0807 07:07:09.768391 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:09.768412 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:09.768766 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:09.769087 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:09.769622 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:09.770908 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:09.772496 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:09.773179 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:09.773305 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:09.774537 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:09.774545 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:09.774559 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:07:09.775077 (+ 518us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:09.775160 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:07:09.775184 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:09.775306 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:09.775559 (+ 253us) server_negotiation.cc:299] Negotiation successful
0807 07:07:09.775730 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":334,"threads_started":1}
I20260807 07:07:09.780341 21836 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 07:07:09.782824 20885 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55a22737589341fe839205c3f3bd8d86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a22737589341fe839205c3f3bd8d86" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42845 } }
I20260807 07:07:09.783296 20885 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55a22737589341fe839205c3f3bd8d86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:07:09.783399 20885 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55a22737589341fe839205c3f3bd8d86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55a22737589341fe839205c3f3bd8d86, State: Initialized, Role: FOLLOWER
I20260807 07:07:09.784091 20885 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55a22737589341fe839205c3f3bd8d86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a22737589341fe839205c3f3bd8d86" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42845 } }
I20260807 07:07:09.784394 20885 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55a22737589341fe839205c3f3bd8d86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:07:09.784555 20885 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55a22737589341fe839205c3f3bd8d86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:07:09.784773 20885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55a22737589341fe839205c3f3bd8d86 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:07:09.787340 20885 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55a22737589341fe839205c3f3bd8d86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a22737589341fe839205c3f3bd8d86" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42845 } }
I20260807 07:07:09.787819 20885 leader_election.cc:302] T 00000000000000000000000000000000 P 55a22737589341fe839205c3f3bd8d86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55a22737589341fe839205c3f3bd8d86; no voters:
I20260807 07:07:09.788445 20885 leader_election.cc:288] T 00000000000000000000000000000000 P 55a22737589341fe839205c3f3bd8d86 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 07:07:09.788549 20898 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55a22737589341fe839205c3f3bd8d86 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:07:09.790723 20885 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55a22737589341fe839205c3f3bd8d86 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:07:09.791617 20898 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55a22737589341fe839205c3f3bd8d86 [term 1 LEADER]: Becoming Leader. State: Replica: 55a22737589341fe839205c3f3bd8d86, State: Running, Role: LEADER
I20260807 07:07:09.792279 20898 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55a22737589341fe839205c3f3bd8d86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a22737589341fe839205c3f3bd8d86" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42845 } }
I20260807 07:07:09.794332 20888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:09.785800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:42349 (local address 127.0.0.1:42845)
0807 07:07:09.785981 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:09.785987 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:09.788002 (+ 2015us) server_negotiation.cc:407] Connection header received
0807 07:07:09.788192 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:09.788198 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:09.788271 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:09.788353 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:09.789013 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:09.789988 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:09.791375 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:09.792069 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:09.792216 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:09.793557 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:09.793563 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:09.793568 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:07:09.793589 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:09.793654 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:07:09.793670 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:09.793768 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:09.793969 (+ 201us) server_negotiation.cc:299] Negotiation successful
0807 07:07:09.794022 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 07:07:09.796432 20900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55a22737589341fe839205c3f3bd8d86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55a22737589341fe839205c3f3bd8d86. Latest consensus state: current_term: 1 leader_uuid: "55a22737589341fe839205c3f3bd8d86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a22737589341fe839205c3f3bd8d86" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42845 } } }
I20260807 07:07:09.796867 20900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55a22737589341fe839205c3f3bd8d86 [sys.catalog]: This master's current role is: LEADER
I20260807 07:07:09.797422 20899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55a22737589341fe839205c3f3bd8d86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55a22737589341fe839205c3f3bd8d86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a22737589341fe839205c3f3bd8d86" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42845 } } }
I20260807 07:07:09.797716 20899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55a22737589341fe839205c3f3bd8d86 [sys.catalog]: This master's current role is: LEADER
I20260807 07:07:09.802265 20913 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:07:09.806761 20913 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:07:09.814600 20913 catalog_manager.cc:1347] Generated new cluster ID: 042885e26c874da793e3ea58544d3e8c
I20260807 07:07:09.814711 20913 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:07:09.837229 20913 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:07:09.838704 20913 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:07:09.847903 20913 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55a22737589341fe839205c3f3bd8d86: Generated new TSK 0
I20260807 07:07:09.849151 20913 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 07:07:09.861677 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 20778
W20260807 07:07:09.862862 20877 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.213:37071
W20260807 07:07:09.863644 21836 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 20778: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 07:07:09.863768 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 20778
W20260807 07:07:09.883597 20758 connection.cc:535] client connection to 127.0.0.1:42845 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42845): Connection reset by peer (error 104)
2026-08-07T07:07:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2073 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786086078080656-21836-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 07:07:09 dist-test-slave-ncp9 krb5kdc[20933](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 07:07:09 dist-test-slave-ncp9 krb5kdc[20933](info): set up 2 sockets
Aug 07 07:07:09 dist-test-slave-ncp9 krb5kdc[20933](info): commencing operation
krb5kdc: starting...
W20260807 07:07:11.078250 21836 mini_kdc.cc:121] Time spent starting KDC: real 1.175s user 0.004s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Aug 07 07:07:11 dist-test-slave-ncp9 krb5kdc[20933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T07:07:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:07:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786086078080656-21836-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786086078080656-21836-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 07:07:11.415361 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39799
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786086078080656-21836-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:07:11.764433 20950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:07:11.764777 20950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:07:11.764951 20950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:07:11.772481 20950 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:07:11.772624 20950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:07:11.772712 20950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:07:11.772794 20950 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:07:11.772856 20950 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:07:11.783279 20950 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39799
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786086078080656-21836-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:07:11.784648 20950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:07:11.786005 20950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:07:11.798010 20956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:07:11.797962 20959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:07:11.798034 20957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:07:11.799130 20950 server_base.cc:1034] running on GCE node
I20260807 07:07:11.799883 20950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:07:11.801381 20950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:07:11.802597 20950 hybrid_clock.cc:648] HybridClock initialized: now 1786086431802548 us; error 34 us; skew 500 ppm
I20260807 07:07:11.802928 20950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:07:11 dist-test-slave-ncp9 krb5kdc[20933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:07:11.810225 20950 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 07:07:11.811765 20950 webserver.cc:460] Webserver started at http://127.0.0.1:42153/ using document root <none> and password file <none>
I20260807 07:07:11.812443 20950 fs_manager.cc:359] Metadata directory not provided
I20260807 07:07:11.812573 20950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:07:11.812836 20950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:07:11.815991 20950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "e0221cdf0c71468ebe67adaefbc79c00"
format_stamp: "Formatted at 2026-08-07 07:07:11 on dist-test-slave-ncp9"
server_key: "30be4f377a99f7bdf9f83fe3b2224ca6"
server_key_iv: "a8c4894fd37165e237e9c3cf21153b3c"
server_key_version: "encryptionkey@0"
I20260807 07:07:11.816756 20950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "e0221cdf0c71468ebe67adaefbc79c00"
format_stamp: "Formatted at 2026-08-07 07:07:11 on dist-test-slave-ncp9"
server_key: "30be4f377a99f7bdf9f83fe3b2224ca6"
server_key_iv: "a8c4894fd37165e237e9c3cf21153b3c"
server_key_version: "encryptionkey@0"
I20260807 07:07:11.822094 20950 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 07:07:11.826043 20971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:07:11.827327 20950 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260807 07:07:11.827491 20950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "e0221cdf0c71468ebe67adaefbc79c00"
format_stamp: "Formatted at 2026-08-07 07:07:11 on dist-test-slave-ncp9"
server_key: "30be4f377a99f7bdf9f83fe3b2224ca6"
server_key_iv: "a8c4894fd37165e237e9c3cf21153b3c"
server_key_version: "encryptionkey@0"
I20260807 07:07:11.827679 20950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:07:11.833289 20950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:07:11.837034 20950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:07:11.837334 20950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:07:11.856320 20950 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39799
I20260807 07:07:11.856367 21046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39799 every 8 connection(s)
I20260807 07:07:11.857722 20950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:07:11.861261 21047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:07:11.861780 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 20950
I20260807 07:07:11.861933 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:07:11.862406 21836 external_mini_cluster.cc:1428] Setting key 1a94651d50b3dd97d3d215c99808668c
I20260807 07:07:11.869972 21052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:11.864027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:44753 (local address 127.0.0.1:39799)
0807 07:07:11.864781 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:11.864792 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:11.864833 (+ 41us) server_negotiation.cc:407] Connection header received
0807 07:07:11.866915 (+ 2082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:11.866937 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:11.867243 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:11.867549 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:11.868531 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:11.868538 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:11.868550 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:07:11.868563 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:11.868634 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:07:11.868658 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:11.868809 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:11.868938 (+ 129us) server_negotiation.cc:299] Negotiation successful
0807 07:07:11.869110 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":325,"threads_started":1}
I20260807 07:07:11.873448 21836 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 07:07:11.873896 21047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0221cdf0c71468ebe67adaefbc79c00: Bootstrap starting.
I20260807 07:07:11.876753 21047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0221cdf0c71468ebe67adaefbc79c00: Neither blocks nor log segments found. Creating new log.
I20260807 07:07:11.877880 21047 log.cc:824] T 00000000000000000000000000000000 P e0221cdf0c71468ebe67adaefbc79c00: Log is configured to *not* fsync() on all Append() calls
I20260807 07:07:11.879803 21052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:11.877535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:56717 (local address 127.0.0.1:39799)
0807 07:07:11.877729 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:11.877735 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:11.878098 (+ 363us) server_negotiation.cc:407] Connection header received
0807 07:07:11.878244 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:11.878247 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:11.878297 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:11.878384 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:11.879219 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:11.879225 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:11.879230 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:07:11.879247 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:11.879283 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:07:11.879296 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:11.879386 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:11.879492 (+ 106us) server_negotiation.cc:299] Negotiation successful
0807 07:07:11.879547 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260807 07:07:11.880635 21047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0221cdf0c71468ebe67adaefbc79c00: No bootstrap required, opened a new log
I20260807 07:07:11.887910 21047 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0221cdf0c71468ebe67adaefbc79c00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0221cdf0c71468ebe67adaefbc79c00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39799 } }
I20260807 07:07:11.888263 21047 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0221cdf0c71468ebe67adaefbc79c00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:07:11.888353 21047 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0221cdf0c71468ebe67adaefbc79c00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0221cdf0c71468ebe67adaefbc79c00, State: Initialized, Role: FOLLOWER
I20260807 07:07:11.888960 21047 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0221cdf0c71468ebe67adaefbc79c00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0221cdf0c71468ebe67adaefbc79c00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39799 } }
I20260807 07:07:11.889179 21047 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0221cdf0c71468ebe67adaefbc79c00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:07:11.889305 21047 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0221cdf0c71468ebe67adaefbc79c00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:07:11.889441 21047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0221cdf0c71468ebe67adaefbc79c00 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:07:11.891551 21047 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0221cdf0c71468ebe67adaefbc79c00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0221cdf0c71468ebe67adaefbc79c00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39799 } }
I20260807 07:07:11.892019 21047 leader_election.cc:302] T 00000000000000000000000000000000 P e0221cdf0c71468ebe67adaefbc79c00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0221cdf0c71468ebe67adaefbc79c00; no voters:
I20260807 07:07:11.892611 21047 leader_election.cc:288] T 00000000000000000000000000000000 P e0221cdf0c71468ebe67adaefbc79c00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 07:07:11.892740 21064 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0221cdf0c71468ebe67adaefbc79c00 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:07:11.893370 21064 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0221cdf0c71468ebe67adaefbc79c00 [term 1 LEADER]: Becoming Leader. State: Replica: e0221cdf0c71468ebe67adaefbc79c00, State: Running, Role: LEADER
I20260807 07:07:11.893934 21064 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0221cdf0c71468ebe67adaefbc79c00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0221cdf0c71468ebe67adaefbc79c00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39799 } }
I20260807 07:07:11.894812 21047 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0221cdf0c71468ebe67adaefbc79c00 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:07:11.896715 21065 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0221cdf0c71468ebe67adaefbc79c00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0221cdf0c71468ebe67adaefbc79c00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0221cdf0c71468ebe67adaefbc79c00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39799 } } }
I20260807 07:07:11.897192 21065 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0221cdf0c71468ebe67adaefbc79c00 [sys.catalog]: This master's current role is: LEADER
I20260807 07:07:11.896704 21067 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0221cdf0c71468ebe67adaefbc79c00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0221cdf0c71468ebe67adaefbc79c00. Latest consensus state: current_term: 1 leader_uuid: "e0221cdf0c71468ebe67adaefbc79c00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0221cdf0c71468ebe67adaefbc79c00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39799 } } }
I20260807 07:07:11.897907 21067 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0221cdf0c71468ebe67adaefbc79c00 [sys.catalog]: This master's current role is: LEADER
I20260807 07:07:11.899314 21078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:07:11.903734 21078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:07:11.912055 21078 catalog_manager.cc:1347] Generated new cluster ID: 0aae79769a6242b5842103d997fc9147
I20260807 07:07:11.912168 21078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:07:11.948519 21078 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:07:11.949040 21078 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:07:11.960529 21078 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0221cdf0c71468ebe67adaefbc79c00: Generated new TSK 0
I20260807 07:07:11.961524 21078 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 07:07:11.968357 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 20950
W20260807 07:07:11.969673 21041 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.213:44753
W20260807 07:07:11.970399 21836 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 20950: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 07:07:11.970547 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 20950
2026-08-07T07:07:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2111 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786086078080656-21836-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 07:07:12 dist-test-slave-ncp9 krb5kdc[21094](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 07:07:12 dist-test-slave-ncp9 krb5kdc[21094](info): set up 2 sockets
Aug 07 07:07:12 dist-test-slave-ncp9 krb5kdc[21094](info): commencing operation
krb5kdc: starting...
W20260807 07:07:13.200886 21836 mini_kdc.cc:121] Time spent starting KDC: real 1.186s user 0.001s sys 0.057s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Aug 07 07:07:13 dist-test-slave-ncp9 krb5kdc[21094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T07:07:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:07:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
I20260807 07:07:13.559901 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:41565
--webserver_interface=127.21.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:40343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:41565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:07:13.912230 21112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:07:13.912525 21112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:07:13.912679 21112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:07:13.920094 21112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:07:13.920202 21112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:07:13.920291 21112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:07:13.920365 21112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:07:13.920425 21112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:07:13.930373 21112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:40343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:41565
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:41565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:07:13.931569 21112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:07:13.932860 21112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:07:13.943837 21118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:07:13.943915 21121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:07:13.943936 21119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:07:13.944962 21112 server_base.cc:1034] running on GCE node
I20260807 07:07:13.945698 21112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:07:13.946995 21112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:07:13.948215 21112 hybrid_clock.cc:648] HybridClock initialized: now 1786086433948164 us; error 44 us; skew 500 ppm
I20260807 07:07:13.948544 21112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:07:13 dist-test-slave-ncp9 krb5kdc[21094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086433, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:07:13.952474 21112 init.cc:377] Logged in from keytab as kudu/127.21.83.62@KRBTEST.COM (short username kudu)
I20260807 07:07:13.954284 21112 webserver.cc:460] Webserver started at http://127.21.83.62:33411/ using document root <none> and password file <none>
I20260807 07:07:13.955015 21112 fs_manager.cc:359] Metadata directory not provided
I20260807 07:07:13.955188 21112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:07:13.955564 21112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:07:13.958871 21112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "ff1f79f0fa174027ac42b21f7168bf28"
format_stamp: "Formatted at 2026-08-07 07:07:13 on dist-test-slave-ncp9"
server_key: "78da67731868b4b61f4eb9fcf827b27c"
server_key_iv: "f6ef19f1d94ac358bdf3dec795064b68"
server_key_version: "encryptionkey@0"
I20260807 07:07:13.959688 21112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "ff1f79f0fa174027ac42b21f7168bf28"
format_stamp: "Formatted at 2026-08-07 07:07:13 on dist-test-slave-ncp9"
server_key: "78da67731868b4b61f4eb9fcf827b27c"
server_key_iv: "f6ef19f1d94ac358bdf3dec795064b68"
server_key_version: "encryptionkey@0"
I20260807 07:07:13.965546 21112 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 07:07:13.969519 21131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:07:13.970969 21112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 07:07:13.971148 21112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "ff1f79f0fa174027ac42b21f7168bf28"
format_stamp: "Formatted at 2026-08-07 07:07:13 on dist-test-slave-ncp9"
server_key: "78da67731868b4b61f4eb9fcf827b27c"
server_key_iv: "f6ef19f1d94ac358bdf3dec795064b68"
server_key_version: "encryptionkey@0"
I20260807 07:07:13.971343 21112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:07:13.993763 21112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:07:13.998126 21112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:07:13.998382 21112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:07:14.017778 21112 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.62:41565
I20260807 07:07:14.017832 21204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.62:41565 every 8 connection(s)
I20260807 07:07:14.019105 21112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:07:14.022900 21205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:07:14.027074 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 21112
I20260807 07:07:14.027235 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:07:14.027856 21836 external_mini_cluster.cc:1428] Setting key 52f04d5932429e9c356493d6d20d9856
I20260807 07:07:14.035224 21205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff1f79f0fa174027ac42b21f7168bf28: Bootstrap starting.
I20260807 07:07:14.039143 21205 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff1f79f0fa174027ac42b21f7168bf28: Neither blocks nor log segments found. Creating new log.
I20260807 07:07:14.040516 21205 log.cc:824] T 00000000000000000000000000000000 P ff1f79f0fa174027ac42b21f7168bf28: Log is configured to *not* fsync() on all Append() calls
Aug 07 07:07:14 dist-test-slave-ncp9 krb5kdc[21094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:07:14.043942 21205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff1f79f0fa174027ac42b21f7168bf28: No bootstrap required, opened a new log
I20260807 07:07:14.053005 21205 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff1f79f0fa174027ac42b21f7168bf28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff1f79f0fa174027ac42b21f7168bf28" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 41565 } }
I20260807 07:07:14.053330 21210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:14.029822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33831 (local address 127.21.83.62:41565)
0807 07:07:14.030656 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:14.030671 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:14.030721 (+ 50us) server_negotiation.cc:407] Connection header received
0807 07:07:14.033368 (+ 2647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:14.033402 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:14.033963 (+ 561us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:14.034422 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:14.035835 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:14.037900 (+ 2065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:14.039226 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:14.040266 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:14.040461 (+ 195us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:14.045387 (+ 4926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:14.045396 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:14.045411 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:07:14.045454 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:14.048757 (+ 3303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:14.049773 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:14.049780 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:14.049787 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:14.049969 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:14.050462 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:14.050466 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:14.050468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:14.051054 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:14.051327 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:14.051842 (+ 515us) server_negotiation.cc:299] Negotiation successful
0807 07:07:14.052189 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":344,"threads_started":1}
I20260807 07:07:14.053411 21205 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff1f79f0fa174027ac42b21f7168bf28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:07:14.053501 21205 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff1f79f0fa174027ac42b21f7168bf28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff1f79f0fa174027ac42b21f7168bf28, State: Initialized, Role: FOLLOWER
I20260807 07:07:14.054149 21205 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff1f79f0fa174027ac42b21f7168bf28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff1f79f0fa174027ac42b21f7168bf28" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 41565 } }
I20260807 07:07:14.054392 21205 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff1f79f0fa174027ac42b21f7168bf28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:07:14.054522 21205 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff1f79f0fa174027ac42b21f7168bf28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:07:14.054706 21205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff1f79f0fa174027ac42b21f7168bf28 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:07:14.057585 21836 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 07:07:14.057796 21205 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff1f79f0fa174027ac42b21f7168bf28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff1f79f0fa174027ac42b21f7168bf28" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 41565 } }
I20260807 07:07:14.058275 21205 leader_election.cc:302] T 00000000000000000000000000000000 P ff1f79f0fa174027ac42b21f7168bf28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff1f79f0fa174027ac42b21f7168bf28; no voters:
I20260807 07:07:14.058853 21205 leader_election.cc:288] T 00000000000000000000000000000000 P ff1f79f0fa174027ac42b21f7168bf28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 07:07:14.059002 21213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff1f79f0fa174027ac42b21f7168bf28 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:07:14.060837 21205 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff1f79f0fa174027ac42b21f7168bf28 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:07:14.069475 21213 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff1f79f0fa174027ac42b21f7168bf28 [term 1 LEADER]: Becoming Leader. State: Replica: ff1f79f0fa174027ac42b21f7168bf28, State: Running, Role: LEADER
I20260807 07:07:14.070156 21213 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff1f79f0fa174027ac42b21f7168bf28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff1f79f0fa174027ac42b21f7168bf28" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 41565 } }
I20260807 07:07:14.072719 21214 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff1f79f0fa174027ac42b21f7168bf28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff1f79f0fa174027ac42b21f7168bf28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff1f79f0fa174027ac42b21f7168bf28" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 41565 } } }
I20260807 07:07:14.073117 21214 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff1f79f0fa174027ac42b21f7168bf28 [sys.catalog]: This master's current role is: LEADER
I20260807 07:07:14.073874 21216 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff1f79f0fa174027ac42b21f7168bf28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff1f79f0fa174027ac42b21f7168bf28. Latest consensus state: current_term: 1 leader_uuid: "ff1f79f0fa174027ac42b21f7168bf28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff1f79f0fa174027ac42b21f7168bf28" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 41565 } } }
I20260807 07:07:14.074342 21216 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff1f79f0fa174027ac42b21f7168bf28 [sys.catalog]: This master's current role is: LEADER
W20260807 07:07:14.084532 21239 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ff1f79f0fa174027ac42b21f7168bf28: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 07:07:14.084661 21239 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 07:07:14.084782 21238 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:07:14.086448 21210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:14.071000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40689 (local address 127.21.83.62:41565)
0807 07:07:14.071186 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:14.071192 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:14.071554 (+ 362us) server_negotiation.cc:407] Connection header received
0807 07:07:14.071736 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:14.071742 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:14.071827 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:14.071932 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:14.073622 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:14.075454 (+ 1832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:14.077079 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:14.077793 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:14.077923 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:14.080211 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:14.080219 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:14.080225 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:07:14.080269 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:14.083438 (+ 3169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:14.084314 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:14.084320 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:14.084323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:14.084472 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:14.085008 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:14.085013 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:14.085016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:14.085402 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:14.085662 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:14.085927 (+ 265us) server_negotiation.cc:299] Negotiation successful
0807 07:07:14.086116 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 07:07:14.086858 21238 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:07:14.094662 21238 catalog_manager.cc:1347] Generated new cluster ID: 43249d55d69c421ea2b2d00a98c19d23
I20260807 07:07:14.094755 21238 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:07:14.103071 21238 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:07:14.104271 21238 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:07:14.114943 21238 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff1f79f0fa174027ac42b21f7168bf28: Generated new TSK 0
I20260807 07:07:14.115859 21238 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 07:07:14.127491 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 21112
I20260807 07:07:14.129160 21198 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:07:14.220789 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 21112
W20260807 07:07:14.243064 21090 connection.cc:535] client connection to 127.21.83.62:41565 recv error: Network error: failed to read from TLS socket (remote: 127.21.83.62:41565): Connection reset by peer (error 104)
2026-08-07T07:07:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2247 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786086078080656-21836-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 07:07:14 dist-test-slave-ncp9 krb5kdc[21251](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 07:07:14 dist-test-slave-ncp9 krb5kdc[21251](info): set up 2 sockets
Aug 07 07:07:14 dist-test-slave-ncp9 krb5kdc[21251](info): commencing operation
krb5kdc: starting...
W20260807 07:07:15.455693 21836 mini_kdc.cc:121] Time spent starting KDC: real 1.193s user 0.000s sys 0.060s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Aug 07 07:07:15 dist-test-slave-ncp9 krb5kdc[21251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T07:07:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:07:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
I20260807 07:07:15.806054 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:39431
--webserver_interface=127.21.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:33365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:39431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:07:16.170181 21275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:07:16.170509 21275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:07:16.170608 21275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:07:16.178045 21275 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:07:16.178174 21275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:07:16.178221 21275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:07:16.178287 21275 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:07:16.178331 21275 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:07:16.188532 21275 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:33365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:39431
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:39431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:07:16.189918 21275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:07:16.191210 21275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:07:16.203115 21288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:07:16.203191 21283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:07:16.203209 21284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:07:16.204378 21275 server_base.cc:1034] running on GCE node
I20260807 07:07:16.205380 21275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:07:16.206869 21275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:07:16.208462 21275 hybrid_clock.cc:648] HybridClock initialized: now 1786086436208376 us; error 46 us; skew 500 ppm
I20260807 07:07:16.209126 21275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:07:16 dist-test-slave-ncp9 krb5kdc[21251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086436, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:07:16.212270 21275 init.cc:377] Logged in from keytab as kudu/127.21.83.62@KRBTEST.COM (short username kudu)
I20260807 07:07:16.213768 21275 webserver.cc:460] Webserver started at http://127.21.83.62:40035/ using document root <none> and password file <none>
I20260807 07:07:16.214421 21275 fs_manager.cc:359] Metadata directory not provided
I20260807 07:07:16.214550 21275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:07:16.214831 21275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:07:16.217841 21275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "0436a4e6ea6044d286c83b4cba7cf134"
format_stamp: "Formatted at 2026-08-07 07:07:16 on dist-test-slave-ncp9"
server_key: "ec9cd9b337447884d7bef64fe2b8d570"
server_key_iv: "78320e8e8d9ed82f99eb21e036e82dec"
server_key_version: "encryptionkey@0"
I20260807 07:07:16.218587 21275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "0436a4e6ea6044d286c83b4cba7cf134"
format_stamp: "Formatted at 2026-08-07 07:07:16 on dist-test-slave-ncp9"
server_key: "ec9cd9b337447884d7bef64fe2b8d570"
server_key_iv: "78320e8e8d9ed82f99eb21e036e82dec"
server_key_version: "encryptionkey@0"
I20260807 07:07:16.223755 21275 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 07:07:16.227135 21311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:07:16.228336 21275 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 07:07:16.228509 21275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "0436a4e6ea6044d286c83b4cba7cf134"
format_stamp: "Formatted at 2026-08-07 07:07:16 on dist-test-slave-ncp9"
server_key: "ec9cd9b337447884d7bef64fe2b8d570"
server_key_iv: "78320e8e8d9ed82f99eb21e036e82dec"
server_key_version: "encryptionkey@0"
I20260807 07:07:16.228683 21275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:07:16.287766 21275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:07:16.292301 21275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:07:16.292621 21275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:07:16.313488 21275 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.62:39431
I20260807 07:07:16.313539 21385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.62:39431 every 8 connection(s)
I20260807 07:07:16.314965 21275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:07:16.318953 21387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:07:16.320047 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 21275
I20260807 07:07:16.320242 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:07:16.320777 21836 external_mini_cluster.cc:1428] Setting key c6b6f3991d6e52aefd94dc65c892ff5a
I20260807 07:07:16.333552 21387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0436a4e6ea6044d286c83b4cba7cf134: Bootstrap starting.
I20260807 07:07:16.335213 21392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:16.322861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40717 (local address 127.21.83.62:39431)
0807 07:07:16.323752 (+ 891us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:16.323768 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:16.323814 (+ 46us) server_negotiation.cc:407] Connection header received
0807 07:07:16.325610 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:16.325638 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:16.326104 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:16.326464 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:16.327115 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:16.328405 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:16.330350 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:16.331132 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:16.331269 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:16.332677 (+ 1408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:16.332689 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:16.332710 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:07:16.333421 (+ 711us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:16.333510 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:07:16.333549 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:16.333685 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:16.333964 (+ 279us) server_negotiation.cc:299] Negotiation successful
0807 07:07:16.334164 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":408,"threads_started":1}
I20260807 07:07:16.337167 21387 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0436a4e6ea6044d286c83b4cba7cf134: Neither blocks nor log segments found. Creating new log.
I20260807 07:07:16.338369 21387 log.cc:824] T 00000000000000000000000000000000 P 0436a4e6ea6044d286c83b4cba7cf134: Log is configured to *not* fsync() on all Append() calls
I20260807 07:07:16.339217 21836 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 07:07:16.341648 21387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0436a4e6ea6044d286c83b4cba7cf134: No bootstrap required, opened a new log
I20260807 07:07:16.350330 21392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:16.344084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35415 (local address 127.21.83.62:39431)
0807 07:07:16.344284 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:16.344290 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:16.344647 (+ 357us) server_negotiation.cc:407] Connection header received
0807 07:07:16.344833 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:16.344837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:16.344941 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:16.345050 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:16.345595 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:16.346368 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:16.347719 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:16.348202 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:16.348313 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:16.349502 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:16.349506 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:16.349510 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:07:16.349530 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:16.349586 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:07:16.349603 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:16.349697 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:16.349882 (+ 185us) server_negotiation.cc:299] Negotiation successful
0807 07:07:16.349937 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 07:07:16.349990 21387 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0436a4e6ea6044d286c83b4cba7cf134 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0436a4e6ea6044d286c83b4cba7cf134" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39431 } }
I20260807 07:07:16.350518 21387 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0436a4e6ea6044d286c83b4cba7cf134 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:07:16.350656 21387 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0436a4e6ea6044d286c83b4cba7cf134 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0436a4e6ea6044d286c83b4cba7cf134, State: Initialized, Role: FOLLOWER
I20260807 07:07:16.351308 21387 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0436a4e6ea6044d286c83b4cba7cf134 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0436a4e6ea6044d286c83b4cba7cf134" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39431 } }
I20260807 07:07:16.351550 21387 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0436a4e6ea6044d286c83b4cba7cf134 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:07:16.351745 21387 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0436a4e6ea6044d286c83b4cba7cf134 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:07:16.351941 21387 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0436a4e6ea6044d286c83b4cba7cf134 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:07:16.354314 21387 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0436a4e6ea6044d286c83b4cba7cf134 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0436a4e6ea6044d286c83b4cba7cf134" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39431 } }
I20260807 07:07:16.354799 21387 leader_election.cc:302] T 00000000000000000000000000000000 P 0436a4e6ea6044d286c83b4cba7cf134 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0436a4e6ea6044d286c83b4cba7cf134; no voters:
I20260807 07:07:16.355361 21387 leader_election.cc:288] T 00000000000000000000000000000000 P 0436a4e6ea6044d286c83b4cba7cf134 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 07:07:16.355542 21407 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0436a4e6ea6044d286c83b4cba7cf134 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:07:16.356652 21407 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0436a4e6ea6044d286c83b4cba7cf134 [term 1 LEADER]: Becoming Leader. State: Replica: 0436a4e6ea6044d286c83b4cba7cf134, State: Running, Role: LEADER
I20260807 07:07:16.357278 21407 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0436a4e6ea6044d286c83b4cba7cf134 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0436a4e6ea6044d286c83b4cba7cf134" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39431 } }
I20260807 07:07:16.357614 21387 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0436a4e6ea6044d286c83b4cba7cf134 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:07:16.360212 21412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0436a4e6ea6044d286c83b4cba7cf134 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0436a4e6ea6044d286c83b4cba7cf134. Latest consensus state: current_term: 1 leader_uuid: "0436a4e6ea6044d286c83b4cba7cf134" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0436a4e6ea6044d286c83b4cba7cf134" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39431 } } }
I20260807 07:07:16.360630 21412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0436a4e6ea6044d286c83b4cba7cf134 [sys.catalog]: This master's current role is: LEADER
I20260807 07:07:16.361159 21408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0436a4e6ea6044d286c83b4cba7cf134 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0436a4e6ea6044d286c83b4cba7cf134" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0436a4e6ea6044d286c83b4cba7cf134" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 39431 } } }
I20260807 07:07:16.361482 21408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0436a4e6ea6044d286c83b4cba7cf134 [sys.catalog]: This master's current role is: LEADER
I20260807 07:07:16.364528 21421 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:07:16.370369 21421 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:07:16.379765 21421 catalog_manager.cc:1347] Generated new cluster ID: 3aa8c21c0d4f4ea68ec618db11e27e0f
I20260807 07:07:16.379916 21421 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:07:16.390800 21421 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:07:16.392820 21421 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:07:16.410878 21421 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0436a4e6ea6044d286c83b4cba7cf134: Generated new TSK 0
I20260807 07:07:16.412122 21421 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 07:07:16.418489 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 21275
W20260807 07:07:16.420014 21380 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40717
W20260807 07:07:16.420843 21836 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 21275: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 07:07:16.421037 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 21275
W20260807 07:07:16.445186 21248 connection.cc:535] client connection to 127.21.83.62:39431 recv error: Network error: failed to read from TLS socket (remote: 127.21.83.62:39431): Connection reset by peer (error 104)
2026-08-07T07:07:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2201 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786086078080656-21836-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 07:07:16 dist-test-slave-ncp9 krb5kdc[21450](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 07:07:16 dist-test-slave-ncp9 krb5kdc[21450](info): set up 2 sockets
Aug 07 07:07:16 dist-test-slave-ncp9 krb5kdc[21450](info): commencing operation
krb5kdc: starting...
W20260807 07:07:17.682379 21836 mini_kdc.cc:121] Time spent starting KDC: real 1.219s user 0.000s sys 0.058s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Aug 07 07:07:17 dist-test-slave-ncp9 krb5kdc[21450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T07:07:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:07:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
I20260807 07:07:18.053202 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:32815
--webserver_interface=127.21.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:34203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:32815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:07:18.407779 21476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:07:18.408144 21476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:07:18.408299 21476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:07:18.416630 21476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:07:18.416778 21476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:07:18.416879 21476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:07:18.416994 21476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:07:18.417049 21476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:07:18.427973 21476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:34203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:32815
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:32815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:07:18.429385 21476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:07:18.430755 21476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:07:18.442636 21483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:07:18.442689 21486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:07:18.442783 21484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:07:18.444352 21476 server_base.cc:1034] running on GCE node
I20260807 07:07:18.445684 21476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:07:18.447882 21476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:07:18.449198 21476 hybrid_clock.cc:648] HybridClock initialized: now 1786086438449120 us; error 49 us; skew 500 ppm
I20260807 07:07:18.449730 21476 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:07:18 dist-test-slave-ncp9 krb5kdc[21450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086438, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:07:18.458165 21476 init.cc:377] Logged in from keytab as kudu/127.21.83.62@KRBTEST.COM (short username kudu)
I20260807 07:07:18.460017 21476 webserver.cc:460] Webserver started at http://127.21.83.62:35567/ using document root <none> and password file <none>
I20260807 07:07:18.460780 21476 fs_manager.cc:359] Metadata directory not provided
I20260807 07:07:18.460934 21476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:07:18.461256 21476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:07:18.464497 21476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "5f0ab2bc8bb84f418f2430198ba3e550"
format_stamp: "Formatted at 2026-08-07 07:07:18 on dist-test-slave-ncp9"
server_key: "97a31a369ddaae35544a1df7f5731e00"
server_key_iv: "189086c00d01c16ec879f9db6fedc296"
server_key_version: "encryptionkey@0"
I20260807 07:07:18.465307 21476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "5f0ab2bc8bb84f418f2430198ba3e550"
format_stamp: "Formatted at 2026-08-07 07:07:18 on dist-test-slave-ncp9"
server_key: "97a31a369ddaae35544a1df7f5731e00"
server_key_iv: "189086c00d01c16ec879f9db6fedc296"
server_key_version: "encryptionkey@0"
I20260807 07:07:18.470402 21476 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260807 07:07:18.473881 21496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:07:18.475178 21476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260807 07:07:18.475351 21476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "5f0ab2bc8bb84f418f2430198ba3e550"
format_stamp: "Formatted at 2026-08-07 07:07:18 on dist-test-slave-ncp9"
server_key: "97a31a369ddaae35544a1df7f5731e00"
server_key_iv: "189086c00d01c16ec879f9db6fedc296"
server_key_version: "encryptionkey@0"
I20260807 07:07:18.475539 21476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:07:18.481199 21476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:07:18.484886 21476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:07:18.485193 21476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:07:18.504390 21476 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.62:32815
I20260807 07:07:18.504429 21570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.62:32815 every 8 connection(s)
I20260807 07:07:18.506124 21476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:07:18.510167 21571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:07:18.511574 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 21476
I20260807 07:07:18.511772 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:07:18.512362 21836 external_mini_cluster.cc:1428] Setting key bd89301cb7f0841f7e6037dddf59342a
I20260807 07:07:18.521196 21575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:18.514539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44395 (local address 127.21.83.62:32815)
0807 07:07:18.515462 (+ 923us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:18.515476 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:18.515532 (+ 56us) server_negotiation.cc:407] Connection header received
0807 07:07:18.517449 (+ 1917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:18.517481 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:18.517888 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:18.518328 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:18.519464 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:18.519478 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:18.519492 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:07:18.519514 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:18.519576 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:07:18.519598 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:18.519813 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:18.519971 (+ 158us) server_negotiation.cc:299] Negotiation successful
0807 07:07:18.520177 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":390,"threads_started":1}
I20260807 07:07:18.525082 21836 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 07:07:18.525497 21571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f0ab2bc8bb84f418f2430198ba3e550: Bootstrap starting.
I20260807 07:07:18.528836 21571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f0ab2bc8bb84f418f2430198ba3e550: Neither blocks nor log segments found. Creating new log.
I20260807 07:07:18.530210 21571 log.cc:824] T 00000000000000000000000000000000 P 5f0ab2bc8bb84f418f2430198ba3e550: Log is configured to *not* fsync() on all Append() calls
I20260807 07:07:18.533105 21575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:18.529547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35877 (local address 127.21.83.62:32815)
0807 07:07:18.529727 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:18.529732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:18.531042 (+ 1310us) server_negotiation.cc:407] Connection header received
0807 07:07:18.531338 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:18.531342 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:18.531397 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:18.531480 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:18.532448 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:18.532452 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:18.532455 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:07:18.532486 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:18.532530 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:07:18.532546 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:18.532638 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:18.532759 (+ 121us) server_negotiation.cc:299] Negotiation successful
0807 07:07:18.532810 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 07:07:18.533865 21571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f0ab2bc8bb84f418f2430198ba3e550: No bootstrap required, opened a new log
I20260807 07:07:18.541646 21571 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f0ab2bc8bb84f418f2430198ba3e550 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f0ab2bc8bb84f418f2430198ba3e550" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 32815 } }
I20260807 07:07:18.542107 21571 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f0ab2bc8bb84f418f2430198ba3e550 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:07:18.542224 21571 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f0ab2bc8bb84f418f2430198ba3e550 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f0ab2bc8bb84f418f2430198ba3e550, State: Initialized, Role: FOLLOWER
I20260807 07:07:18.543175 21571 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f0ab2bc8bb84f418f2430198ba3e550 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f0ab2bc8bb84f418f2430198ba3e550" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 32815 } }
I20260807 07:07:18.543725 21571 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f0ab2bc8bb84f418f2430198ba3e550 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:07:18.543895 21571 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f0ab2bc8bb84f418f2430198ba3e550 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:07:18.544083 21571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f0ab2bc8bb84f418f2430198ba3e550 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:07:18.546542 21571 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f0ab2bc8bb84f418f2430198ba3e550 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f0ab2bc8bb84f418f2430198ba3e550" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 32815 } }
I20260807 07:07:18.547148 21571 leader_election.cc:302] T 00000000000000000000000000000000 P 5f0ab2bc8bb84f418f2430198ba3e550 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f0ab2bc8bb84f418f2430198ba3e550; no voters:
I20260807 07:07:18.547789 21571 leader_election.cc:288] T 00000000000000000000000000000000 P 5f0ab2bc8bb84f418f2430198ba3e550 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 07:07:18.547940 21586 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f0ab2bc8bb84f418f2430198ba3e550 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:07:18.549006 21586 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f0ab2bc8bb84f418f2430198ba3e550 [term 1 LEADER]: Becoming Leader. State: Replica: 5f0ab2bc8bb84f418f2430198ba3e550, State: Running, Role: LEADER
I20260807 07:07:18.549638 21586 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f0ab2bc8bb84f418f2430198ba3e550 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f0ab2bc8bb84f418f2430198ba3e550" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 32815 } }
I20260807 07:07:18.550190 21571 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f0ab2bc8bb84f418f2430198ba3e550 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:07:18.552312 21587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0ab2bc8bb84f418f2430198ba3e550 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f0ab2bc8bb84f418f2430198ba3e550" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f0ab2bc8bb84f418f2430198ba3e550" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 32815 } } }
I20260807 07:07:18.552731 21587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0ab2bc8bb84f418f2430198ba3e550 [sys.catalog]: This master's current role is: LEADER
I20260807 07:07:18.553548 21588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0ab2bc8bb84f418f2430198ba3e550 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f0ab2bc8bb84f418f2430198ba3e550. Latest consensus state: current_term: 1 leader_uuid: "5f0ab2bc8bb84f418f2430198ba3e550" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f0ab2bc8bb84f418f2430198ba3e550" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 32815 } } }
I20260807 07:07:18.553936 21588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0ab2bc8bb84f418f2430198ba3e550 [sys.catalog]: This master's current role is: LEADER
I20260807 07:07:18.555078 21593 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:07:18.561173 21593 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:07:18.571714 21593 catalog_manager.cc:1347] Generated new cluster ID: 1ae40b5cfbee4da5a430e8fe10e3cebf
I20260807 07:07:18.571887 21593 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:07:18.591503 21593 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:07:18.591982 21593 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:07:18.599694 21593 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f0ab2bc8bb84f418f2430198ba3e550: Generated new TSK 0
I20260807 07:07:18.600593 21593 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 07:07:18.611322 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 21476
W20260807 07:07:18.612557 21564 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44395
W20260807 07:07:18.613368 21836 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 21476: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 07:07:18.613533 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 21476
2026-08-07T07:07:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2193 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786086078080656-21836-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 07:07:18 dist-test-slave-ncp9 krb5kdc[21612](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 07:07:18 dist-test-slave-ncp9 krb5kdc[21612](info): set up 2 sockets
Aug 07 07:07:18 dist-test-slave-ncp9 krb5kdc[21612](info): commencing operation
krb5kdc: starting...
W20260807 07:07:19.859584 21836 mini_kdc.cc:121] Time spent starting KDC: real 1.202s user 0.000s sys 0.062s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Aug 07 07:07:20 dist-test-slave-ncp9 krb5kdc[21612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T07:07:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:07:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
I20260807 07:07:20.224270 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:33209
--webserver_interface=127.21.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:45519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:33209
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:07:20.571461 21632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:07:20.571744 21632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:07:20.571866 21632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:07:20.579075 21632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:07:20.579186 21632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:07:20.579259 21632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:07:20.579344 21632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:07:20.579407 21632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:07:20.589162 21632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:45519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:33209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:33209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:07:20.590487 21632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:07:20.591743 21632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:07:20.603399 21638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:07:20.603471 21646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:07:20.603442 21644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:07:20.604611 21632 server_base.cc:1034] running on GCE node
I20260807 07:07:20.605506 21632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:07:20.606998 21632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:07:20.608230 21632 hybrid_clock.cc:648] HybridClock initialized: now 1786086440608179 us; error 40 us; skew 500 ppm
I20260807 07:07:20.608752 21632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:07:20 dist-test-slave-ncp9 krb5kdc[21612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086440, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:07:20.612298 21632 init.cc:377] Logged in from keytab as kudu/127.21.83.62@KRBTEST.COM (short username kudu)
I20260807 07:07:20.613989 21632 webserver.cc:460] Webserver started at http://127.21.83.62:33959/ using document root <none> and password file <none>
I20260807 07:07:20.614805 21632 fs_manager.cc:359] Metadata directory not provided
I20260807 07:07:20.614954 21632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:07:20.615268 21632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:07:20.618490 21632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "6c7e9443c065420a924cb957c34e2179"
format_stamp: "Formatted at 2026-08-07 07:07:20 on dist-test-slave-ncp9"
server_key: "966de7b485eed063cb711c9036dab957"
server_key_iv: "df9b314ea8bfa03c9dcb90fdc8da53e6"
server_key_version: "encryptionkey@0"
I20260807 07:07:20.619297 21632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "6c7e9443c065420a924cb957c34e2179"
format_stamp: "Formatted at 2026-08-07 07:07:20 on dist-test-slave-ncp9"
server_key: "966de7b485eed063cb711c9036dab957"
server_key_iv: "df9b314ea8bfa03c9dcb90fdc8da53e6"
server_key_version: "encryptionkey@0"
I20260807 07:07:20.624859 21632 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 07:07:20.628404 21655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:07:20.629814 21632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 07:07:20.629999 21632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "6c7e9443c065420a924cb957c34e2179"
format_stamp: "Formatted at 2026-08-07 07:07:20 on dist-test-slave-ncp9"
server_key: "966de7b485eed063cb711c9036dab957"
server_key_iv: "df9b314ea8bfa03c9dcb90fdc8da53e6"
server_key_version: "encryptionkey@0"
I20260807 07:07:20.630208 21632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:07:20.670521 21632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:07:20.674618 21632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:07:20.674921 21632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:07:20.693877 21632 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.62:33209
I20260807 07:07:20.693944 21739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.62:33209 every 8 connection(s)
I20260807 07:07:20.695282 21632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:07:20.699066 21740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:07:20.702322 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 21632
I20260807 07:07:20.702517 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:07:20.703043 21836 external_mini_cluster.cc:1428] Setting key bc47cd9eafc4fa49e15b36ba1cf0937d
I20260807 07:07:20.711234 21740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c7e9443c065420a924cb957c34e2179: Bootstrap starting.
I20260807 07:07:20.714991 21740 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c7e9443c065420a924cb957c34e2179: Neither blocks nor log segments found. Creating new log.
I20260807 07:07:20.716954 21740 log.cc:824] T 00000000000000000000000000000000 P 6c7e9443c065420a924cb957c34e2179: Log is configured to *not* fsync() on all Append() calls
Aug 07 07:07:20 dist-test-slave-ncp9 krb5kdc[21612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:07:20.720808 21740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c7e9443c065420a924cb957c34e2179: No bootstrap required, opened a new log
I20260807 07:07:20.730089 21744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:20.705043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50315 (local address 127.21.83.62:33209)
0807 07:07:20.705894 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:20.705906 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:20.705945 (+ 39us) server_negotiation.cc:407] Connection header received
0807 07:07:20.707614 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:20.707637 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:20.708018 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:20.708372 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:20.710541 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:20.712462 (+ 1921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:20.714172 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:20.715038 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:20.715242 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:20.720169 (+ 4927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:20.720181 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:20.720203 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:07:20.720255 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:20.724583 (+ 4328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:20.725588 (+ 1005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:20.725599 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:20.725610 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:20.725867 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:20.726329 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:20.726334 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:20.726338 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:20.727195 (+ 857us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:20.727552 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:20.727985 (+ 433us) server_negotiation.cc:299] Negotiation successful
0807 07:07:20.728527 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":405,"threads_started":1}
I20260807 07:07:20.731805 21740 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c7e9443c065420a924cb957c34e2179 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c7e9443c065420a924cb957c34e2179" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 33209 } }
I20260807 07:07:20.732321 21740 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c7e9443c065420a924cb957c34e2179 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:07:20.732435 21740 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c7e9443c065420a924cb957c34e2179 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c7e9443c065420a924cb957c34e2179, State: Initialized, Role: FOLLOWER
I20260807 07:07:20.733115 21740 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c7e9443c065420a924cb957c34e2179 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c7e9443c065420a924cb957c34e2179" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 33209 } }
I20260807 07:07:20.733400 21740 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c7e9443c065420a924cb957c34e2179 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:07:20.733539 21740 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c7e9443c065420a924cb957c34e2179 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:07:20.733731 21740 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c7e9443c065420a924cb957c34e2179 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:07:20.734668 21836 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 07:07:20.736317 21740 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c7e9443c065420a924cb957c34e2179 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c7e9443c065420a924cb957c34e2179" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 33209 } }
I20260807 07:07:20.736848 21740 leader_election.cc:302] T 00000000000000000000000000000000 P 6c7e9443c065420a924cb957c34e2179 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c7e9443c065420a924cb957c34e2179; no voters:
I20260807 07:07:20.737531 21740 leader_election.cc:288] T 00000000000000000000000000000000 P 6c7e9443c065420a924cb957c34e2179 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 07:07:20.737651 21747 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c7e9443c065420a924cb957c34e2179 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:07:20.739135 21747 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c7e9443c065420a924cb957c34e2179 [term 1 LEADER]: Becoming Leader. State: Replica: 6c7e9443c065420a924cb957c34e2179, State: Running, Role: LEADER
I20260807 07:07:20.739777 21747 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c7e9443c065420a924cb957c34e2179 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c7e9443c065420a924cb957c34e2179" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 33209 } }
I20260807 07:07:20.739982 21740 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c7e9443c065420a924cb957c34e2179 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:07:20.746927 21755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c7e9443c065420a924cb957c34e2179 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c7e9443c065420a924cb957c34e2179. Latest consensus state: current_term: 1 leader_uuid: "6c7e9443c065420a924cb957c34e2179" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c7e9443c065420a924cb957c34e2179" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 33209 } } }
I20260807 07:07:20.746834 21751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c7e9443c065420a924cb957c34e2179 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c7e9443c065420a924cb957c34e2179" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c7e9443c065420a924cb957c34e2179" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 33209 } } }
I20260807 07:07:20.747234 21751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c7e9443c065420a924cb957c34e2179 [sys.catalog]: This master's current role is: LEADER
I20260807 07:07:20.747234 21755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c7e9443c065420a924cb957c34e2179 [sys.catalog]: This master's current role is: LEADER
I20260807 07:07:20.749420 21764 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:07:20.755051 21764 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:07:20.761904 21744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:20.740123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60719 (local address 127.21.83.62:33209)
0807 07:07:20.740293 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:20.740299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:20.742167 (+ 1868us) server_negotiation.cc:407] Connection header received
0807 07:07:20.742396 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:20.742402 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:20.742507 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:20.742621 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:20.744594 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:20.745629 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:20.749905 (+ 4276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:20.750708 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:20.750873 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:20.753577 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:20.753583 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:20.753586 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:07:20.753624 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:20.756820 (+ 3196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:20.757866 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:20.757873 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:20.757877 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:20.758084 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:20.758815 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:20.758820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:20.758828 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:20.759354 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:20.759659 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:20.761180 (+ 1521us) server_negotiation.cc:299] Negotiation successful
0807 07:07:20.761468 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260807 07:07:20.771093 21764 catalog_manager.cc:1347] Generated new cluster ID: 415417d4ab5143b6ac4f8b1897e27988
I20260807 07:07:20.771227 21764 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:07:20.787642 21764 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:07:20.789543 21764 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:07:20.810211 21764 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c7e9443c065420a924cb957c34e2179: Generated new TSK 0
I20260807 07:07:20.811247 21764 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 07:07:20.824761 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 21632
I20260807 07:07:20.825996 21731 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:07:20.921924 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 21632
2026-08-07T07:07:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2307 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786086078080656-21836-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 07:07:21 dist-test-slave-ncp9 krb5kdc[21786](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 07:07:21 dist-test-slave-ncp9 krb5kdc[21786](info): set up 2 sockets
Aug 07 07:07:21 dist-test-slave-ncp9 krb5kdc[21786](info): commencing operation
krb5kdc: starting...
W20260807 07:07:22.153249 21836 mini_kdc.cc:121] Time spent starting KDC: real 1.188s user 0.000s sys 0.057s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Aug 07 07:07:22 dist-test-slave-ncp9 krb5kdc[21786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T07:07:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:07:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
I20260807 07:07:22.497701 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:46365
--webserver_interface=127.21.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:41967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:46365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:07:22.843237 21804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:07:22.843560 21804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:07:22.843685 21804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:07:22.851275 21804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:07:22.851418 21804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:07:22.851471 21804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:07:22.851559 21804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:07:22.851615 21804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:07:22.862046 21804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:41967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:46365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:46365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:07:22.863404 21804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:07:22.864758 21804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:07:22.876008 21816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:07:22.875999 21814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:07:22.876364 21812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:07:22.876936 21804 server_base.cc:1034] running on GCE node
I20260807 07:07:22.877693 21804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:07:22.879077 21804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:07:22.880283 21804 hybrid_clock.cc:648] HybridClock initialized: now 1786086442880238 us; error 40 us; skew 500 ppm
I20260807 07:07:22.880604 21804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:07:22 dist-test-slave-ncp9 krb5kdc[21786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086442, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:07:22.883591 21804 init.cc:377] Logged in from keytab as kudu/127.21.83.62@KRBTEST.COM (short username kudu)
I20260807 07:07:22.884965 21804 webserver.cc:460] Webserver started at http://127.21.83.62:44913/ using document root <none> and password file <none>
I20260807 07:07:22.885605 21804 fs_manager.cc:359] Metadata directory not provided
I20260807 07:07:22.885705 21804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:07:22.885953 21804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:07:22.888814 21804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "6cf52f1c278c4fbb9827e416194082e6"
format_stamp: "Formatted at 2026-08-07 07:07:22 on dist-test-slave-ncp9"
server_key: "854ebe091279f8b08708d17c95a0efdd"
server_key_iv: "0291f0d87905a02944ddb8b7d158a1de"
server_key_version: "encryptionkey@0"
I20260807 07:07:22.889556 21804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "6cf52f1c278c4fbb9827e416194082e6"
format_stamp: "Formatted at 2026-08-07 07:07:22 on dist-test-slave-ncp9"
server_key: "854ebe091279f8b08708d17c95a0efdd"
server_key_iv: "0291f0d87905a02944ddb8b7d158a1de"
server_key_version: "encryptionkey@0"
I20260807 07:07:22.894375 21804 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 07:07:22.897641 21824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:07:22.898953 21804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 07:07:22.899116 21804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "6cf52f1c278c4fbb9827e416194082e6"
format_stamp: "Formatted at 2026-08-07 07:07:22 on dist-test-slave-ncp9"
server_key: "854ebe091279f8b08708d17c95a0efdd"
server_key_iv: "0291f0d87905a02944ddb8b7d158a1de"
server_key_version: "encryptionkey@0"
I20260807 07:07:22.899289 21804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:07:22.912776 21804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:07:22.916731 21804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:07:22.917022 21804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:07:22.934713 21804 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.62:46365
I20260807 07:07:22.934743 21920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.62:46365 every 8 connection(s)
I20260807 07:07:22.936108 21804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:07:22.939921 21921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:07:22.944106 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 21804
I20260807 07:07:22.944301 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:07:22.944836 21836 external_mini_cluster.cc:1428] Setting key af6494233853d29aad22fb56bf8ac5f7
I20260807 07:07:22.952831 21921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cf52f1c278c4fbb9827e416194082e6: Bootstrap starting.
I20260807 07:07:22.956310 21921 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cf52f1c278c4fbb9827e416194082e6: Neither blocks nor log segments found. Creating new log.
I20260807 07:07:22.957476 21921 log.cc:824] T 00000000000000000000000000000000 P 6cf52f1c278c4fbb9827e416194082e6: Log is configured to *not* fsync() on all Append() calls
I20260807 07:07:22.957813 21924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:22.946634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56333 (local address 127.21.83.62:46365)
0807 07:07:22.947373 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:22.947385 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:22.947427 (+ 42us) server_negotiation.cc:407] Connection header received
0807 07:07:22.949090 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:22.949113 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:22.949546 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:22.949871 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:22.950451 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:22.951764 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:22.953507 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:22.954171 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:22.954319 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:22.955580 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:22.955587 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:22.955599 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:07:22.956107 (+ 508us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:22.956193 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:07:22.956220 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:22.956313 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:22.956555 (+ 242us) server_negotiation.cc:299] Negotiation successful
0807 07:07:22.956812 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":318,"threads_started":1}
I20260807 07:07:22.960474 21921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cf52f1c278c4fbb9827e416194082e6: No bootstrap required, opened a new log
I20260807 07:07:22.961634 21836 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 07:07:22.970916 21921 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cf52f1c278c4fbb9827e416194082e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cf52f1c278c4fbb9827e416194082e6" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 46365 } }
I20260807 07:07:22.971465 21921 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6cf52f1c278c4fbb9827e416194082e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:07:22.971588 21921 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cf52f1c278c4fbb9827e416194082e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cf52f1c278c4fbb9827e416194082e6, State: Initialized, Role: FOLLOWER
I20260807 07:07:22.972311 21921 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cf52f1c278c4fbb9827e416194082e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cf52f1c278c4fbb9827e416194082e6" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 46365 } }
I20260807 07:07:22.972620 21921 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6cf52f1c278c4fbb9827e416194082e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:07:22.972775 21921 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cf52f1c278c4fbb9827e416194082e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:07:22.972999 21921 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cf52f1c278c4fbb9827e416194082e6 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:07:22.973930 21924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:22.967173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35753 (local address 127.21.83.62:46365)
0807 07:07:22.967395 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:22.967401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:22.967730 (+ 329us) server_negotiation.cc:407] Connection header received
0807 07:07:22.967918 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:22.967926 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:22.968001 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:22.968093 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:22.968690 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:22.969519 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:22.971285 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:22.971836 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:22.971957 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:22.973176 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:22.973183 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:22.973189 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:07:22.973214 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:22.973261 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:07:22.973279 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:22.973363 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:22.973518 (+ 155us) server_negotiation.cc:299] Negotiation successful
0807 07:07:22.973601 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260807 07:07:22.975936 21921 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cf52f1c278c4fbb9827e416194082e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cf52f1c278c4fbb9827e416194082e6" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 46365 } }
I20260807 07:07:22.976446 21921 leader_election.cc:302] T 00000000000000000000000000000000 P 6cf52f1c278c4fbb9827e416194082e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6cf52f1c278c4fbb9827e416194082e6; no voters:
I20260807 07:07:22.977064 21921 leader_election.cc:288] T 00000000000000000000000000000000 P 6cf52f1c278c4fbb9827e416194082e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 07:07:22.977200 21938 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cf52f1c278c4fbb9827e416194082e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:07:22.977761 21938 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6cf52f1c278c4fbb9827e416194082e6 [term 1 LEADER]: Becoming Leader. State: Replica: 6cf52f1c278c4fbb9827e416194082e6, State: Running, Role: LEADER
I20260807 07:07:22.978404 21938 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cf52f1c278c4fbb9827e416194082e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cf52f1c278c4fbb9827e416194082e6" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 46365 } }
I20260807 07:07:22.980482 21938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cf52f1c278c4fbb9827e416194082e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cf52f1c278c4fbb9827e416194082e6. Latest consensus state: current_term: 1 leader_uuid: "6cf52f1c278c4fbb9827e416194082e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cf52f1c278c4fbb9827e416194082e6" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 46365 } } }
I20260807 07:07:22.980816 21938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cf52f1c278c4fbb9827e416194082e6 [sys.catalog]: This master's current role is: LEADER
I20260807 07:07:22.981074 21941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cf52f1c278c4fbb9827e416194082e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6cf52f1c278c4fbb9827e416194082e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cf52f1c278c4fbb9827e416194082e6" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 46365 } } }
I20260807 07:07:22.981465 21941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cf52f1c278c4fbb9827e416194082e6 [sys.catalog]: This master's current role is: LEADER
I20260807 07:07:22.982085 21921 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cf52f1c278c4fbb9827e416194082e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:07:22.982261 21943 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:07:22.987969 21943 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:07:22.998414 21943 catalog_manager.cc:1347] Generated new cluster ID: abaf2157771449e5bc58f2735f08dac3
I20260807 07:07:22.998543 21943 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:07:23.015439 21943 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:07:23.017300 21943 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:07:23.024017 21943 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6cf52f1c278c4fbb9827e416194082e6: Generated new TSK 0
I20260807 07:07:23.024848 21943 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 07:07:23.039156 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 21804
W20260807 07:07:23.040467 21915 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56333
W20260807 07:07:23.041224 21836 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 21804: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 07:07:23.041404 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 21804
2026-08-07T07:07:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2123 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786086078080656-21836-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 07:07:23 dist-test-slave-ncp9 krb5kdc[21965](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 07:07:23 dist-test-slave-ncp9 krb5kdc[21965](info): set up 2 sockets
Aug 07 07:07:23 dist-test-slave-ncp9 krb5kdc[21965](info): commencing operation
krb5kdc: starting...
W20260807 07:07:24.293817 21836 mini_kdc.cc:121] Time spent starting KDC: real 1.206s user 0.000s sys 0.060s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Aug 07 07:07:24 dist-test-slave-ncp9 krb5kdc[21965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T07:07:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:07:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
I20260807 07:07:24.671829 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:42251
--webserver_interface=127.21.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:42597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:42251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:07:25.019217 21984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:07:25.019503 21984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:07:25.019625 21984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:07:25.026708 21984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:07:25.026820 21984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:07:25.026898 21984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:07:25.027001 21984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:07:25.027060 21984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:07:25.036844 21984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:42597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:42251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:42251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:07:25.038249 21984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:07:25.039798 21984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:07:25.052030 21993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:07:25.052007 21999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:07:25.052086 21995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:07:25.053469 21984 server_base.cc:1034] running on GCE node
I20260807 07:07:25.054368 21984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:07:25.055938 21984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:07:25.057153 21984 hybrid_clock.cc:648] HybridClock initialized: now 1786086445057117 us; error 34 us; skew 500 ppm
I20260807 07:07:25.057513 21984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:07:25 dist-test-slave-ncp9 krb5kdc[21965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086445, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:07:25.064716 21984 init.cc:377] Logged in from keytab as kudu/127.21.83.62@KRBTEST.COM (short username kudu)
I20260807 07:07:25.066279 21984 webserver.cc:460] Webserver started at http://127.21.83.62:40925/ using document root <none> and password file <none>
I20260807 07:07:25.066985 21984 fs_manager.cc:359] Metadata directory not provided
I20260807 07:07:25.067106 21984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:07:25.067389 21984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:07:25.070370 21984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "194b2086bed7447fbf051735c4e3602a"
format_stamp: "Formatted at 2026-08-07 07:07:25 on dist-test-slave-ncp9"
server_key: "823b9c5b8ca300f086e6157408fb74c3"
server_key_iv: "db7482fad2594744cfc0fa46e64251f7"
server_key_version: "encryptionkey@0"
I20260807 07:07:25.071116 21984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "194b2086bed7447fbf051735c4e3602a"
format_stamp: "Formatted at 2026-08-07 07:07:25 on dist-test-slave-ncp9"
server_key: "823b9c5b8ca300f086e6157408fb74c3"
server_key_iv: "db7482fad2594744cfc0fa46e64251f7"
server_key_version: "encryptionkey@0"
I20260807 07:07:25.076022 21984 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260807 07:07:25.079250 22010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:07:25.080487 21984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 07:07:25.080659 21984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "194b2086bed7447fbf051735c4e3602a"
format_stamp: "Formatted at 2026-08-07 07:07:25 on dist-test-slave-ncp9"
server_key: "823b9c5b8ca300f086e6157408fb74c3"
server_key_iv: "db7482fad2594744cfc0fa46e64251f7"
server_key_version: "encryptionkey@0"
I20260807 07:07:25.080838 21984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:07:25.086378 21984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:07:25.090258 21984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:07:25.090535 21984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:07:25.108246 21984 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.62:42251
I20260807 07:07:25.108289 22093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.62:42251 every 8 connection(s)
I20260807 07:07:25.109707 21984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:07:25.113323 22094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:07:25.117491 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 21984
I20260807 07:07:25.117679 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:07:25.118217 21836 external_mini_cluster.cc:1428] Setting key a811b671a6892adaaccc3f5e22d15ee9
I20260807 07:07:25.124467 22094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 194b2086bed7447fbf051735c4e3602a: Bootstrap starting.
I20260807 07:07:25.126777 22099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:25.119920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51569 (local address 127.21.83.62:42251)
0807 07:07:25.120703 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:25.120714 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:25.120754 (+ 40us) server_negotiation.cc:407] Connection header received
0807 07:07:25.123084 (+ 2330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:25.123109 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:25.123431 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:25.123749 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:25.125147 (+ 1398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:25.125156 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:25.125200 (+ 44us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:07:25.125222 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:25.125315 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:07:25.125345 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:25.125580 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:25.125711 (+ 131us) server_negotiation.cc:299] Negotiation successful
0807 07:07:25.125894 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":355,"threads_started":1}
I20260807 07:07:25.127776 22094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 194b2086bed7447fbf051735c4e3602a: Neither blocks nor log segments found. Creating new log.
I20260807 07:07:25.129132 22094 log.cc:824] T 00000000000000000000000000000000 P 194b2086bed7447fbf051735c4e3602a: Log is configured to *not* fsync() on all Append() calls
I20260807 07:07:25.130769 21836 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 07:07:25.132212 22094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 194b2086bed7447fbf051735c4e3602a: No bootstrap required, opened a new log
I20260807 07:07:25.138062 22099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:25.135229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58307 (local address 127.21.83.62:42251)
0807 07:07:25.135463 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:25.135471 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:25.135879 (+ 408us) server_negotiation.cc:407] Connection header received
0807 07:07:25.136109 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:25.136114 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:25.136179 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:25.136271 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:25.137380 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:25.137385 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:25.137388 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 07:07:25.137406 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:25.137453 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 07:07:25.137474 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:25.137576 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:25.137692 (+ 116us) server_negotiation.cc:299] Negotiation successful
0807 07:07:25.137750 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 07:07:25.141037 22094 raft_consensus.cc:348] T 00000000000000000000000000000000 P 194b2086bed7447fbf051735c4e3602a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "194b2086bed7447fbf051735c4e3602a" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 42251 } }
I20260807 07:07:25.141474 22094 raft_consensus.cc:374] T 00000000000000000000000000000000 P 194b2086bed7447fbf051735c4e3602a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:07:25.141569 22094 raft_consensus.cc:729] T 00000000000000000000000000000000 P 194b2086bed7447fbf051735c4e3602a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 194b2086bed7447fbf051735c4e3602a, State: Initialized, Role: FOLLOWER
I20260807 07:07:25.142174 22094 consensus_queue.cc:260] T 00000000000000000000000000000000 P 194b2086bed7447fbf051735c4e3602a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "194b2086bed7447fbf051735c4e3602a" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 42251 } }
I20260807 07:07:25.142452 22094 raft_consensus.cc:388] T 00000000000000000000000000000000 P 194b2086bed7447fbf051735c4e3602a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:07:25.142593 22094 raft_consensus.cc:482] T 00000000000000000000000000000000 P 194b2086bed7447fbf051735c4e3602a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:07:25.142764 22094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 194b2086bed7447fbf051735c4e3602a [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:07:25.145144 22094 raft_consensus.cc:504] T 00000000000000000000000000000000 P 194b2086bed7447fbf051735c4e3602a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "194b2086bed7447fbf051735c4e3602a" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 42251 } }
I20260807 07:07:25.145726 22094 leader_election.cc:302] T 00000000000000000000000000000000 P 194b2086bed7447fbf051735c4e3602a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 194b2086bed7447fbf051735c4e3602a; no voters:
I20260807 07:07:25.146348 22094 leader_election.cc:288] T 00000000000000000000000000000000 P 194b2086bed7447fbf051735c4e3602a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 07:07:25.146489 22120 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 194b2086bed7447fbf051735c4e3602a [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:07:25.147161 22120 raft_consensus.cc:686] T 00000000000000000000000000000000 P 194b2086bed7447fbf051735c4e3602a [term 1 LEADER]: Becoming Leader. State: Replica: 194b2086bed7447fbf051735c4e3602a, State: Running, Role: LEADER
I20260807 07:07:25.147754 22120 consensus_queue.cc:237] T 00000000000000000000000000000000 P 194b2086bed7447fbf051735c4e3602a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "194b2086bed7447fbf051735c4e3602a" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 42251 } }
I20260807 07:07:25.148485 22094 sys_catalog.cc:564] T 00000000000000000000000000000000 P 194b2086bed7447fbf051735c4e3602a [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:07:25.150553 22121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 194b2086bed7447fbf051735c4e3602a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "194b2086bed7447fbf051735c4e3602a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "194b2086bed7447fbf051735c4e3602a" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 42251 } } }
I20260807 07:07:25.150993 22121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 194b2086bed7447fbf051735c4e3602a [sys.catalog]: This master's current role is: LEADER
I20260807 07:07:25.151481 22122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 194b2086bed7447fbf051735c4e3602a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 194b2086bed7447fbf051735c4e3602a. Latest consensus state: current_term: 1 leader_uuid: "194b2086bed7447fbf051735c4e3602a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "194b2086bed7447fbf051735c4e3602a" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 42251 } } }
I20260807 07:07:25.151800 22122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 194b2086bed7447fbf051735c4e3602a [sys.catalog]: This master's current role is: LEADER
I20260807 07:07:25.151970 22125 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:07:25.158845 22125 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:07:25.168637 22125 catalog_manager.cc:1347] Generated new cluster ID: f3de3f21bd214391894d0c6bbdd38433
I20260807 07:07:25.168787 22125 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:07:25.190140 22125 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:07:25.190578 22125 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:07:25.213858 22125 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 194b2086bed7447fbf051735c4e3602a: Generated new TSK 0
I20260807 07:07:25.215049 22125 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 07:07:25.228084 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 21984
W20260807 07:07:25.229355 22088 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51569
W20260807 07:07:25.230110 21836 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 21984: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 07:07:25.230242 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 21984
2026-08-07T07:07:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2190 ms)
[----------] 18 tests from AuthTokenIssuingTest (39443 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 07:07:25 dist-test-slave-ncp9 krb5kdc[22148](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 07:07:25 dist-test-slave-ncp9 krb5kdc[22148](info): set up 2 sockets
Aug 07 07:07:25 dist-test-slave-ncp9 krb5kdc[22148](info): commencing operation
krb5kdc: starting...
W20260807 07:07:26.486049 21836 mini_kdc.cc:121] Time spent starting KDC: real 1.207s user 0.003s sys 0.059s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Aug 07 07:07:26 dist-test-slave-ncp9 krb5kdc[22148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T07:07:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:07:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
I20260807 07:07:26.870440 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:44033
--webserver_interface=127.21.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:37137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:44033,127.21.83.61:39897,127.21.83.60:42159
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:07:27.225013 22168 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 07:07:27.225314 22168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:07:27.225383 22168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:07:27.225464 22168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:07:27.232625 22168 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:07:27.232738 22168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:07:27.232805 22168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:07:27.232856 22168 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:07:27.232949 22168 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:07:27.243285 22168 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:37137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:44033,127.21.83.61:39897,127.21.83.60:42159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:44033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:07:27.244625 22168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:07:27.246012 22168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:07:27.257874 22178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:07:27.257972 22177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:07:27.257967 22180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:07:27.258750 22168 server_base.cc:1034] running on GCE node
I20260807 07:07:27.259567 22168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:07:27.261003 22168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:07:27.262231 22168 hybrid_clock.cc:648] HybridClock initialized: now 1786086447262155 us; error 38 us; skew 500 ppm
I20260807 07:07:27.262573 22168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:07:27 dist-test-slave-ncp9 krb5kdc[22148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086447, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:07:27.265760 22168 init.cc:377] Logged in from keytab as kudu/127.21.83.62@KRBTEST.COM (short username kudu)
I20260807 07:07:27.267246 22168 webserver.cc:460] Webserver started at http://127.21.83.62:39705/ using document root <none> and password file <none>
I20260807 07:07:27.267915 22168 fs_manager.cc:359] Metadata directory not provided
I20260807 07:07:27.268002 22168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:07:27.268275 22168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:07:27.271380 22168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "7161ad4f43d048e489907b04512c1417"
format_stamp: "Formatted at 2026-08-07 07:07:27 on dist-test-slave-ncp9"
server_key: "d73b63fe384764b9abd9cea3ff4b3fb3"
server_key_iv: "d0e8d6b32d715a1c9d497d0e63ff0630"
server_key_version: "encryptionkey@0"
I20260807 07:07:27.272146 22168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "7161ad4f43d048e489907b04512c1417"
format_stamp: "Formatted at 2026-08-07 07:07:27 on dist-test-slave-ncp9"
server_key: "d73b63fe384764b9abd9cea3ff4b3fb3"
server_key_iv: "d0e8d6b32d715a1c9d497d0e63ff0630"
server_key_version: "encryptionkey@0"
I20260807 07:07:27.277359 22168 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260807 07:07:27.280936 22187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:07:27.282291 22168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 07:07:27.282459 22168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "7161ad4f43d048e489907b04512c1417"
format_stamp: "Formatted at 2026-08-07 07:07:27 on dist-test-slave-ncp9"
server_key: "d73b63fe384764b9abd9cea3ff4b3fb3"
server_key_iv: "d0e8d6b32d715a1c9d497d0e63ff0630"
server_key_version: "encryptionkey@0"
I20260807 07:07:27.282662 22168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:07:27.303673 22168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:07:27.307500 22168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:07:27.307749 22168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:07:27.325297 22168 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.62:44033
I20260807 07:07:27.325326 22270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.62:44033 every 8 connection(s)
I20260807 07:07:27.326545 22168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:07:27.328490 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 22168
I20260807 07:07:27.328668 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:07:27.329375 21836 external_mini_cluster.cc:1428] Setting key fd1149d4126d4e9381f3e489d5611599
I20260807 07:07:27.330184 22271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:07:27.336499 22271 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.83.62" port: 44033 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 07:07:27 dist-test-slave-ncp9 krb5kdc[22148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086447, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:07:27.359194 22274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:27.339528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40960 (local address 127.21.83.62:44033)
0807 07:07:27.340325 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:27.340342 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:27.340756 (+ 414us) server_negotiation.cc:407] Connection header received
0807 07:07:27.342696 (+ 1940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:27.342702 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:27.342900 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:27.343087 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:27.344327 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:27.345562 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:27.346910 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:27.347473 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:27.347645 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:27.351488 (+ 3843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:27.351499 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:27.351513 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:07:27.351551 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:27.354446 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:27.355380 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:27.355387 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:27.355389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:27.355563 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:27.355959 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:27.355964 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:27.355966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:27.356531 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:27.356730 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:27.357378 (+ 648us) server_negotiation.cc:299] Negotiation successful
0807 07:07:27.357708 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":569,"thread_start_us":356,"threads_started":1}
I20260807 07:07:27.359234 22275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:27.339528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:44033 (local address 127.0.0.1:40960)
0807 07:07:27.340601 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:27.340640 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0807 07:07:27.342348 (+ 1708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:07:27.343113 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:07:27.343119 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:07:27.343804 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:07:27.344131 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:07:27.344147 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:07:27.345706 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:07:27.345710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:07:27.346771 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:07:27.346784 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:07:27.347707 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:07:27.347715 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:07:27.347942 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:27.348618 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:07:27.348647 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:07:27.351230 (+ 2583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:07:27.354721 (+ 3491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:07:27.354734 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:07:27.354745 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:07:27.355235 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:07:27.355678 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:07:27.355684 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:07:27.355687 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:07:27.355854 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:07:27.356674 (+ 820us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:07:27.356683 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:07:27.356970 (+ 287us) client_negotiation.cc:769] Sending connection context
0807 07:07:27.357232 (+ 262us) client_negotiation.cc:240] Negotiation successful
0807 07:07:27.357853 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":863,"mutex_wait_us":193,"thread_start_us":662,"threads_started":1}
I20260807 07:07:27.362484 22271 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39897 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.21.83.61@KRBTEST.COM; defaulting to no policy
I20260807 07:07:27.363539 22275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:27.363150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.61:39897 (local address 127.0.0.1:43390)
0807 07:07:27.363328 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:27.363425 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.61:39897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260807 07:07:27.363754 22190 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.83.61:39897: connect: Connection refused (error 111)
I20260807 07:07:27.365124 22275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:27.364758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.61:39897 (local address 127.0.0.1:43400)
0807 07:07:27.364924 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:27.364998 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.61:39897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260807 07:07:27.365310 22271 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.83.61:39897: Network error: Client connection negotiation failed: client connection to 127.21.83.61:39897: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.61.keytab.
Entry for principal kudu/127.21.83.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.61.keytab.
I20260807 07:07:27.403311 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.61:39897
--webserver_interface=127.21.83.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.61.keytab
--principal=kudu/127.21.83.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:37137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:44033,127.21.83.61:39897,127.21.83.60:42159
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 07:07:27.414644 22271 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39897 } attempt: 1
I20260807 07:07:27.415779 22275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:27.415346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.61:39897 (local address 127.0.0.1:43404)
0807 07:07:27.415542 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:27.415634 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.61:39897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260807 07:07:27.416821 22275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:27.416451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.61:39897 (local address 127.0.0.1:43414)
0807 07:07:27.416624 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:27.416701 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.61:39897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260807 07:07:27.417034 22271 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.83.61:39897: Network error: Client connection negotiation failed: client connection to 127.21.83.61:39897: connect: Connection refused (error 111)
I20260807 07:07:27.485327 22271 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39897 } attempt: 2
I20260807 07:07:27.486586 22275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:27.486110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.61:39897 (local address 127.0.0.1:43426)
0807 07:07:27.486334 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:27.486435 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.61:39897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260807 07:07:27.487808 22275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:27.487401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.61:39897 (local address 127.0.0.1:43442)
0807 07:07:27.487577 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:27.487668 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.61:39897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260807 07:07:27.488026 22271 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.83.61:39897: Network error: Client connection negotiation failed: client connection to 127.21.83.61:39897: connect: Connection refused (error 111)
I20260807 07:07:27.579264 22271 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39897 } attempt: 3
I20260807 07:07:27.580461 22275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:27.579981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.61:39897 (local address 127.0.0.1:43450)
0807 07:07:27.580206 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:27.580301 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.61:39897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260807 07:07:27.581440 22275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:27.581115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.61:39897 (local address 127.0.0.1:43462)
0807 07:07:27.581235 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:27.581304 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.61:39897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260807 07:07:27.581655 22271 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.83.61:39897: Network error: Client connection negotiation failed: client connection to 127.21.83.61:39897: connect: Connection refused (error 111)
I20260807 07:07:27.725044 22271 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39897 } attempt: 4
I20260807 07:07:27.726397 22275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:27.725800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.61:39897 (local address 127.0.0.1:43468)
0807 07:07:27.726168 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:27.726261 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.61:39897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195}
I20260807 07:07:27.727437 22275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:27.727075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.61:39897 (local address 127.0.0.1:43484)
0807 07:07:27.727211 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:27.727300 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.61:39897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260807 07:07:27.727635 22271 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.83.61:39897: Network error: Client connection negotiation failed: client connection to 127.21.83.61:39897: connect: Connection refused (error 111)
W20260807 07:07:27.748207 22281 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 07:07:27.748548 22281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:07:27.748648 22281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:07:27.748781 22281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:07:27.756204 22281 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:07:27.756304 22281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:07:27.756371 22281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:07:27.756443 22281 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:07:27.756510 22281 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:07:27.766501 22281 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:37137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:44033,127.21.83.61:39897,127.21.83.60:42159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.61.keytab
--principal=kudu/127.21.83.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.61:39897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.83.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:07:27.767725 22281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:07:27.768997 22281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:07:27.779933 22289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:07:27.780032 22294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:07:27.780000 22288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:07:27.780962 22281 server_base.cc:1034] running on GCE node
I20260807 07:07:27.781638 22281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:07:27.782827 22281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:07:27.784034 22281 hybrid_clock.cc:648] HybridClock initialized: now 1786086447783970 us; error 39 us; skew 500 ppm
I20260807 07:07:27.784319 22281 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:07:27 dist-test-slave-ncp9 krb5kdc[22148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086447, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:07:27.787305 22281 init.cc:377] Logged in from keytab as kudu/127.21.83.61@KRBTEST.COM (short username kudu)
I20260807 07:07:27.788589 22281 webserver.cc:460] Webserver started at http://127.21.83.61:40131/ using document root <none> and password file <none>
I20260807 07:07:27.789250 22281 fs_manager.cc:359] Metadata directory not provided
I20260807 07:07:27.789342 22281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:07:27.789638 22281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:07:27.792330 22281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/minicluster-data/master-1/data/instance:
uuid: "767dbfd3946043968c601e97c4bfd1e2"
format_stamp: "Formatted at 2026-08-07 07:07:27 on dist-test-slave-ncp9"
server_key: "90f8c46370304e4b78fb30c1582b0e0b"
server_key_iv: "6e3f0fa9ae8c60eb20dd2a36d8318a33"
server_key_version: "encryptionkey@0"
I20260807 07:07:27.793097 22281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/minicluster-data/master-1/wal/instance:
uuid: "767dbfd3946043968c601e97c4bfd1e2"
format_stamp: "Formatted at 2026-08-07 07:07:27 on dist-test-slave-ncp9"
server_key: "90f8c46370304e4b78fb30c1582b0e0b"
server_key_iv: "6e3f0fa9ae8c60eb20dd2a36d8318a33"
server_key_version: "encryptionkey@0"
I20260807 07:07:27.797654 22281 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260807 07:07:27.800676 22303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:07:27.801855 22281 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260807 07:07:27.801994 22281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/minicluster-data/master-1/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/minicluster-data/master-1/wal
uuid: "767dbfd3946043968c601e97c4bfd1e2"
format_stamp: "Formatted at 2026-08-07 07:07:27 on dist-test-slave-ncp9"
server_key: "90f8c46370304e4b78fb30c1582b0e0b"
server_key_iv: "6e3f0fa9ae8c60eb20dd2a36d8318a33"
server_key_version: "encryptionkey@0"
I20260807 07:07:27.802176 22281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:07:27.816427 22281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:07:27.819820 22281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:07:27.820032 22281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:07:27.836520 22281 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.61:39897
I20260807 07:07:27.836542 22391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.61:39897 every 8 connection(s)
I20260807 07:07:27.837689 22281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/minicluster-data/master-1/data/info.pb
I20260807 07:07:27.840860 22392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:07:27.847188 22392 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.83.62" port: 44033 } has no permanent_uuid. Determining permanent_uuid...
I20260807 07:07:27.848196 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 22281
I20260807 07:07:27.848382 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/minicluster-data/master-1/wal/instance
I20260807 07:07:27.848971 21836 external_mini_cluster.cc:1428] Setting key bad2ee495a1a646152d11aeb72012421
Aug 07 07:07:27 dist-test-slave-ncp9 krb5kdc[22148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086447, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.61@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:07:27.872282 22274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:27.850964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40976 (local address 127.21.83.62:44033)
0807 07:07:27.851195 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:27.851202 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:27.852503 (+ 1301us) server_negotiation.cc:407] Connection header received
0807 07:07:27.854626 (+ 2123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:27.854632 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:27.854723 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:27.854837 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:27.857179 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:27.858072 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:27.859859 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:27.860426 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:27.860551 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:27.864674 (+ 4123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:27.864682 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:27.864686 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:07:27.864732 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:27.867579 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:27.868840 (+ 1261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:27.868845 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:27.868848 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:27.869064 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:27.869781 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:27.869789 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:27.869792 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:27.870323 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:27.870597 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:27.871777 (+ 1180us) server_negotiation.cc:299] Negotiation successful
0807 07:07:27.872022 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260807 07:07:27.873724 22394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:27.851420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:44033 (local address 127.0.0.1:40976)
0807 07:07:27.852300 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:27.852352 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0807 07:07:27.854306 (+ 1954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:07:27.855444 (+ 1138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:07:27.855456 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:07:27.856228 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:07:27.856949 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:07:27.856979 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:07:27.858299 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:07:27.858303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:07:27.859669 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:07:27.859686 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:07:27.860711 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:07:27.860717 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:07:27.861007 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:27.861760 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:07:27.861786 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:07:27.864373 (+ 2587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:07:27.867778 (+ 3405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:07:27.867789 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:07:27.867809 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:07:27.868509 (+ 700us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:07:27.869209 (+ 700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:07:27.869214 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:07:27.869217 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:07:27.869528 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:07:27.870580 (+ 1052us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:07:27.870588 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:07:27.871205 (+ 617us) client_negotiation.cc:769] Sending connection context
0807 07:07:27.871563 (+ 358us) client_negotiation.cc:240] Negotiation successful
0807 07:07:27.872141 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":367,"threads_started":1}
I20260807 07:07:27.875795 22392 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39897 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 07:07:27 dist-test-slave-ncp9 krb5kdc[22148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086447, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.61@KRBTEST.COM for kudu/127.21.83.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.83.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.60@KRBTEST.COM" created.
I20260807 07:07:27.891125 22394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:27.876689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.61:39897 (local address 127.0.0.1:43500)
0807 07:07:27.876869 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:27.876885 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 07:07:27.877025 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:07:27.878220 (+ 1195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:07:27.878225 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:07:27.878716 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:07:27.878863 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:07:27.878873 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:07:27.880287 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:07:27.880291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:07:27.881167 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:07:27.881188 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:07:27.882049 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:07:27.882055 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:07:27.882235 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:27.882852 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:07:27.882870 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:07:27.885197 (+ 2327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:07:27.888490 (+ 3293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:07:27.888494 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:07:27.888497 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:07:27.888950 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:07:27.889337 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:07:27.889343 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:07:27.889347 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:07:27.889557 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:07:27.890336 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:07:27.890340 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:07:27.890482 (+ 142us) client_negotiation.cc:769] Sending connection context
0807 07:07:27.890599 (+ 117us) client_negotiation.cc:240] Negotiation successful
0807 07:07:27.890819 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260807 07:07:27.891461 22399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:27.877044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43500 (local address 127.21.83.61:39897)
0807 07:07:27.877765 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:27.877772 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:27.877798 (+ 26us) server_negotiation.cc:407] Connection header received
0807 07:07:27.877848 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:27.877854 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:27.878051 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:27.878201 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:27.879034 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:27.880143 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:27.881306 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:27.881881 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:27.881996 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:27.885416 (+ 3420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:27.885430 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:27.885445 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:07:27.885476 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:27.888336 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:27.889076 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:27.889085 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:27.889088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:27.889232 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:27.889701 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:27.889706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:27.889709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:27.890199 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:27.890403 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:27.890781 (+ 378us) server_negotiation.cc:299] Negotiation successful
0807 07:07:27.891048 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":408,"threads_started":1}
I20260807 07:07:27.893873 22392 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.83.60" port: 42159 } has no permanent_uuid. Determining permanent_uuid...
I20260807 07:07:27.895149 22394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:27.894662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:42159 (local address 127.0.0.1:54692)
0807 07:07:27.894844 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:27.894989 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260807 07:07:27.895483 22309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.83.60:42159: connect: Connection refused (error 111)
I20260807 07:07:27.897126 22394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:27.896751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:42159 (local address 127.0.0.1:54704)
0807 07:07:27.896922 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:27.897012 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260807 07:07:27.897418 22392 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.83.60:42159: Network error: Client connection negotiation failed: client connection to 127.21.83.60:42159: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.60.keytab.
Entry for principal kudu/127.21.83.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.60.keytab.
I20260807 07:07:27.930078 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.60:42159
--webserver_interface=127.21.83.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.60.keytab
--principal=kudu/127.21.83.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:37137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:44033,127.21.83.61:39897,127.21.83.60:42159
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 07:07:27.946769 22392 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.83.60" port: 42159 } attempt: 1
I20260807 07:07:27.948091 22394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:27.947572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:42159 (local address 127.0.0.1:54712)
0807 07:07:27.947809 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:27.947925 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260807 07:07:27.949247 22394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:27.948854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:42159 (local address 127.0.0.1:54724)
0807 07:07:27.949046 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:27.949128 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260807 07:07:27.949467 22392 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.83.60:42159: Network error: Client connection negotiation failed: client connection to 127.21.83.60:42159: connect: Connection refused (error 111)
I20260807 07:07:28.017675 22392 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.83.60" port: 42159 } attempt: 2
I20260807 07:07:28.018751 22394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:28.018308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:42159 (local address 127.0.0.1:54728)
0807 07:07:28.018503 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:28.018594 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260807 07:07:28.019801 22394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:28.019460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:42159 (local address 127.0.0.1:54736)
0807 07:07:28.019607 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:28.019677 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260807 07:07:28.019994 22392 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.83.60:42159: Network error: Client connection negotiation failed: client connection to 127.21.83.60:42159: connect: Connection refused (error 111)
I20260807 07:07:28.026889 22271 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39897 } attempt: 5
Aug 07 07:07:28 dist-test-slave-ncp9 krb5kdc[22148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086447, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for kudu/127.21.83.61@KRBTEST.COM
I20260807 07:07:28.039408 22275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:28.027524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.61:39897 (local address 127.0.0.1:43514)
0807 07:07:28.027696 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:28.027716 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 07:07:28.027839 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:07:28.028185 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:07:28.028190 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:07:28.028587 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:07:28.028770 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:07:28.028778 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:07:28.029797 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:07:28.029801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:07:28.030674 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:07:28.030685 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:07:28.031414 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:07:28.031419 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:07:28.031612 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:28.032185 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:07:28.032203 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:07:28.034203 (+ 2000us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:07:28.037121 (+ 2918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:07:28.037126 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:07:28.037130 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:07:28.037485 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:07:28.037825 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:07:28.037830 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:07:28.037833 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:07:28.037961 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:07:28.038470 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:07:28.038475 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:07:28.038623 (+ 148us) client_negotiation.cc:769] Sending connection context
0807 07:07:28.038773 (+ 150us) client_negotiation.cc:240] Negotiation successful
0807 07:07:28.039050 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260807 07:07:28.039410 22399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:28.027635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43514 (local address 127.21.83.61:39897)
0807 07:07:28.027801 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:28.027806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:28.027827 (+ 21us) server_negotiation.cc:407] Connection header received
0807 07:07:28.027958 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:28.027963 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:28.028053 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:28.028135 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:28.028938 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:28.029655 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:28.030814 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:28.031270 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:28.031394 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:28.034385 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:28.034389 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:28.034402 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:07:28.034434 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:28.036993 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:28.037593 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:28.037599 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:28.037603 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:28.037722 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:28.038078 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:28.038082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:28.038084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:28.038357 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:28.038485 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:28.038888 (+ 403us) server_negotiation.cc:299] Negotiation successful
0807 07:07:28.039113 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260807 07:07:28.040553 22271 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.83.60" port: 42159 } has no permanent_uuid. Determining permanent_uuid...
I20260807 07:07:28.041694 22275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:28.041322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:42159 (local address 127.0.0.1:54748)
0807 07:07:28.041480 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:28.041563 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260807 07:07:28.042559 22275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:28.042259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:42159 (local address 127.0.0.1:54750)
0807 07:07:28.042392 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:28.042463 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260807 07:07:28.042737 22271 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.83.60:42159: Network error: Client connection negotiation failed: client connection to 127.21.83.60:42159: connect: Connection refused (error 111)
I20260807 07:07:28.094039 22271 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.83.60" port: 42159 } attempt: 1
I20260807 07:07:28.095340 22275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:28.094811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:42159 (local address 127.0.0.1:54752)
0807 07:07:28.095055 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:28.095152 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260807 07:07:28.096341 22275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:28.096009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:42159 (local address 127.0.0.1:54758)
0807 07:07:28.096163 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:28.096233 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260807 07:07:28.096552 22271 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.83.60:42159: Network error: Client connection negotiation failed: client connection to 127.21.83.60:42159: connect: Connection refused (error 111)
I20260807 07:07:28.111209 22392 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.83.60" port: 42159 } attempt: 3
I20260807 07:07:28.112293 22394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:28.111828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:42159 (local address 127.0.0.1:54766)
0807 07:07:28.112037 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:28.112139 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260807 07:07:28.113488 22394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:28.113054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:42159 (local address 127.0.0.1:54772)
0807 07:07:28.113278 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:28.113360 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
W20260807 07:07:28.113703 22392 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.83.60:42159: Network error: Client connection negotiation failed: client connection to 127.21.83.60:42159: connect: Connection refused (error 111)
I20260807 07:07:28.164808 22271 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.83.60" port: 42159 } attempt: 2
I20260807 07:07:28.166018 22275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:28.165511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:42159 (local address 127.0.0.1:54778)
0807 07:07:28.165741 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:28.165846 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260807 07:07:28.166971 22275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:28.166634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:42159 (local address 127.0.0.1:54780)
0807 07:07:28.166778 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:28.166848 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260807 07:07:28.167181 22271 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.83.60:42159: Network error: Client connection negotiation failed: client connection to 127.21.83.60:42159: connect: Connection refused (error 111)
I20260807 07:07:28.256976 22392 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.83.60" port: 42159 } attempt: 4
I20260807 07:07:28.258095 22394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:28.257611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:42159 (local address 127.0.0.1:54792)
0807 07:07:28.257849 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:28.257936 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260807 07:07:28.259249 22394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:28.258885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:42159 (local address 127.0.0.1:54798)
0807 07:07:28.259057 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:28.259132 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260807 07:07:28.259454 22392 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.83.60:42159: Network error: Client connection negotiation failed: client connection to 127.21.83.60:42159: connect: Connection refused (error 111)
I20260807 07:07:28.273398 22271 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.83.60" port: 42159 } attempt: 3
I20260807 07:07:28.274501 22275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:28.274056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:42159 (local address 127.0.0.1:54810)
0807 07:07:28.274250 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:28.274341 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260807 07:07:28.275496 22275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:28.275092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:42159 (local address 127.0.0.1:54824)
0807 07:07:28.275289 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:28.275365 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260807 07:07:28.275683 22271 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.83.60:42159: Network error: Client connection negotiation failed: client connection to 127.21.83.60:42159: connect: Connection refused (error 111)
W20260807 07:07:28.278187 22403 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 07:07:28.278542 22403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:07:28.278640 22403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:07:28.278756 22403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:07:28.286666 22403 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:07:28.286782 22403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:07:28.286875 22403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:07:28.286940 22403 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:07:28.287039 22403 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:07:28.297547 22403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:37137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:44033,127.21.83.61:39897,127.21.83.60:42159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.60.keytab
--principal=kudu/127.21.83.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.60:42159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.83.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:07:28.298856 22403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:07:28.300098 22403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:07:28.311414 22410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:07:28.311367 22413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:07:28.311462 22415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:07:28.312515 22403 server_base.cc:1034] running on GCE node
I20260807 07:07:28.313288 22403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:07:28.314592 22403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:07:28.315793 22403 hybrid_clock.cc:648] HybridClock initialized: now 1786086448315734 us; error 35 us; skew 500 ppm
I20260807 07:07:28.316098 22403 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:07:28 dist-test-slave-ncp9 krb5kdc[22148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086448, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:07:28.319105 22403 init.cc:377] Logged in from keytab as kudu/127.21.83.60@KRBTEST.COM (short username kudu)
I20260807 07:07:28.320520 22403 webserver.cc:460] Webserver started at http://127.21.83.60:45169/ using document root <none> and password file <none>
I20260807 07:07:28.321241 22403 fs_manager.cc:359] Metadata directory not provided
I20260807 07:07:28.321346 22403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:07:28.321645 22403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:07:28.324553 22403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/minicluster-data/master-2/data/instance:
uuid: "7326015bc4ac4e4191d3fd7debacaa4e"
format_stamp: "Formatted at 2026-08-07 07:07:28 on dist-test-slave-ncp9"
server_key: "7e0bddba80c1fa353500b7dac5137a3e"
server_key_iv: "c5d649829957ba838fa73e52ceec613e"
server_key_version: "encryptionkey@0"
I20260807 07:07:28.325344 22403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/minicluster-data/master-2/wal/instance:
uuid: "7326015bc4ac4e4191d3fd7debacaa4e"
format_stamp: "Formatted at 2026-08-07 07:07:28 on dist-test-slave-ncp9"
server_key: "7e0bddba80c1fa353500b7dac5137a3e"
server_key_iv: "c5d649829957ba838fa73e52ceec613e"
server_key_version: "encryptionkey@0"
I20260807 07:07:28.330377 22403 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260807 07:07:28.333725 22424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:07:28.334980 22403 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 07:07:28.335156 22403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/minicluster-data/master-2/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/minicluster-data/master-2/wal
uuid: "7326015bc4ac4e4191d3fd7debacaa4e"
format_stamp: "Formatted at 2026-08-07 07:07:28 on dist-test-slave-ncp9"
server_key: "7e0bddba80c1fa353500b7dac5137a3e"
server_key_iv: "c5d649829957ba838fa73e52ceec613e"
server_key_version: "encryptionkey@0"
I20260807 07:07:28.335366 22403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:07:28.352706 22403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:07:28.357223 22403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:07:28.357499 22403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:07:28.376487 22403 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.60:42159
I20260807 07:07:28.376498 22501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.60:42159 every 8 connection(s)
I20260807 07:07:28.377918 22403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/minicluster-data/master-2/data/info.pb
I20260807 07:07:28.381866 22502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:07:28.387442 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 22403
I20260807 07:07:28.387624 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786086078080656-21836-0/minicluster-data/master-2/wal/instance
I20260807 07:07:28.388057 21836 external_mini_cluster.cc:1428] Setting key 5421f790aaebd01f1f2a9df0ef395014
I20260807 07:07:28.389235 22502 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.83.62" port: 44033 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 07:07:28 dist-test-slave-ncp9 krb5kdc[22148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086448, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.60@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
Aug 07 07:07:28 dist-test-slave-ncp9 krb5kdc[22148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.60@KRBTEST.COM
I20260807 07:07:28.407322 22513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:28.392244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40988 (local address 127.21.83.62:44033)
0807 07:07:28.393681 (+ 1437us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:28.393686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:28.393707 (+ 21us) server_negotiation.cc:407] Connection header received
0807 07:07:28.393890 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:28.393894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:28.393965 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:28.394061 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:28.395317 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:28.396093 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:28.397590 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:28.397996 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:28.398103 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:28.401584 (+ 3481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:28.401588 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:28.401592 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:07:28.401622 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:28.404325 (+ 2703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:28.405068 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:28.405072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:28.405074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:28.405190 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:28.405611 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:28.405614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:28.405616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:28.405894 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:28.406031 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:28.406690 (+ 659us) server_negotiation.cc:299] Negotiation successful
0807 07:07:28.406911 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1329,"thread_start_us":1199,"threads_started":1}
I20260807 07:07:28.408335 22507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:28.390068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54828 (local address 127.21.83.60:42159)
0807 07:07:28.390883 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:28.390899 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:28.390958 (+ 59us) server_negotiation.cc:407] Connection header received
0807 07:07:28.393630 (+ 2672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:28.393653 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:28.393886 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:28.394075 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:28.395577 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:28.396738 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:28.397994 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:28.398549 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:28.398673 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:28.402524 (+ 3851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:28.402532 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:28.402544 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:07:28.402577 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:28.405369 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:28.406034 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:28.406040 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:28.406042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:28.406169 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:28.406563 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:28.406567 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:28.406569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:28.406962 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:28.407264 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:28.407564 (+ 300us) server_negotiation.cc:299] Negotiation successful
0807 07:07:28.407801 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":577,"thread_start_us":348,"threads_started":1}
I20260807 07:07:28.408354 22509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:28.392238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:44033 (local address 127.0.0.1:40988)
0807 07:07:28.393041 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:28.393082 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 07:07:28.393641 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:07:28.394138 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:07:28.394145 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:07:28.394801 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:07:28.395160 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:07:28.395177 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:07:28.396243 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:07:28.396247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:07:28.397452 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:07:28.397465 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:07:28.398216 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:07:28.398228 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:07:28.398463 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:28.399128 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:07:28.399157 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:07:28.401411 (+ 2254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:07:28.404439 (+ 3028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:07:28.404450 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:07:28.404461 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:07:28.404958 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:07:28.405292 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:07:28.405297 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:07:28.405300 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:07:28.405498 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:07:28.406011 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:07:28.406019 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:07:28.406324 (+ 305us) client_negotiation.cc:769] Sending connection context
0807 07:07:28.406581 (+ 257us) client_negotiation.cc:240] Negotiation successful
0807 07:07:28.407095 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":72,"thread_start_us":352,"threads_started":1}
I20260807 07:07:28.409968 22502 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39897 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 07:07:28 dist-test-slave-ncp9 krb5kdc[22148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086448, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.60@KRBTEST.COM for kudu/127.21.83.61@KRBTEST.COM
Aug 07 07:07:28 dist-test-slave-ncp9 krb5kdc[22148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:07:28.424973 22509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:28.410738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.61:39897 (local address 127.0.0.1:43526)
0807 07:07:28.410909 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:28.410928 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 07:07:28.411032 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:07:28.411380 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:07:28.411385 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:07:28.411837 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:07:28.412019 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:07:28.412031 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:07:28.413383 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:07:28.413388 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:07:28.414708 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:07:28.414719 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:07:28.415681 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:07:28.415686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:07:28.415839 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:28.416354 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:07:28.416377 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:07:28.418792 (+ 2415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:07:28.422392 (+ 3600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:07:28.422398 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:07:28.422403 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:07:28.422814 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:07:28.423262 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:07:28.423267 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:07:28.423270 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:07:28.423447 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:07:28.424158 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:07:28.424162 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:07:28.424309 (+ 147us) client_negotiation.cc:769] Sending connection context
0807 07:07:28.424435 (+ 126us) client_negotiation.cc:240] Negotiation successful
0807 07:07:28.424682 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260807 07:07:28.425299 22399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:28.410673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43526 (local address 127.21.83.61:39897)
0807 07:07:28.410863 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:28.410869 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:28.410990 (+ 121us) server_negotiation.cc:407] Connection header received
0807 07:07:28.411158 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:28.411165 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:28.411252 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:28.411350 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:28.412173 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:28.413208 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:28.414858 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:28.415525 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:28.415656 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:28.419027 (+ 3371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:28.419034 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:28.419040 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:07:28.419091 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:28.422251 (+ 3160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:28.422946 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:28.422953 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:28.422957 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:28.423149 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:28.423574 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:28.423580 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:28.423584 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:28.423986 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:28.424173 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:28.424587 (+ 414us) server_negotiation.cc:299] Negotiation successful
0807 07:07:28.424827 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260807 07:07:28.426756 22502 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.83.60" port: 42159 } has no permanent_uuid. Determining permanent_uuid...
I20260807 07:07:28.428942 22513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:28.411345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40998 (local address 127.21.83.62:44033)
0807 07:07:28.411498 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:28.411504 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:28.411621 (+ 117us) server_negotiation.cc:407] Connection header received
0807 07:07:28.411738 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:28.411742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:28.411821 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:28.411916 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:28.413383 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:28.414426 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:28.416058 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:28.416587 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:28.416711 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:28.421751 (+ 5040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:28.421756 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:28.421760 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:07:28.421794 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:28.425313 (+ 3519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:28.426697 (+ 1384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:28.426704 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:28.426707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:28.426897 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:28.427452 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:28.427458 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:28.427461 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:28.427835 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:28.428127 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:28.428488 (+ 361us) server_negotiation.cc:299] Negotiation successful
0807 07:07:28.428689 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Aug 07 07:07:28 dist-test-slave-ncp9 krb5kdc[22148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086448, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.60@KRBTEST.COM for kudu/127.21.83.60@KRBTEST.COM
Aug 07 07:07:28 dist-test-slave-ncp9 krb5kdc[22148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.61@KRBTEST.COM
I20260807 07:07:28.440850 22509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:28.427601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:42159 (local address 127.0.0.1:54838)
0807 07:07:28.427770 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:28.427789 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 07:07:28.427886 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:07:28.428277 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:07:28.428281 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:07:28.428783 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:07:28.429013 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:07:28.429026 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:07:28.430165 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:07:28.430170 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:07:28.431285 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:07:28.431298 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:07:28.431917 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:07:28.431921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:07:28.432092 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:28.432629 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:07:28.432646 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:07:28.434791 (+ 2145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:07:28.438424 (+ 3633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:07:28.438428 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:07:28.438431 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:07:28.438836 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:07:28.439204 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:07:28.439207 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:07:28.439209 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:07:28.439333 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:07:28.439877 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:07:28.439881 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:07:28.440033 (+ 152us) client_negotiation.cc:769] Sending connection context
0807 07:07:28.440153 (+ 120us) client_negotiation.cc:240] Negotiation successful
0807 07:07:28.440445 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260807 07:07:28.440914 22507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:28.427820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54838 (local address 127.21.83.60:42159)
0807 07:07:28.427992 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:28.427997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:28.428013 (+ 16us) server_negotiation.cc:407] Connection header received
0807 07:07:28.428073 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:28.428078 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:28.428144 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:28.428223 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:28.429190 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:28.430011 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:28.431407 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:28.431793 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:28.431889 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:28.434977 (+ 3088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:28.434983 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:28.434989 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:07:28.435024 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:28.438284 (+ 3260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:28.438942 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:28.438947 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:28.438951 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:28.439100 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:28.439432 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:28.439436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:28.439438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:28.439776 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:28.439904 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:28.440254 (+ 350us) server_negotiation.cc:299] Negotiation successful
0807 07:07:28.440610 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":135,"server-negotiator.queue_time_us":60}
I20260807 07:07:28.445503 22502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7326015bc4ac4e4191d3fd7debacaa4e: Bootstrap starting.
I20260807 07:07:28.446126 22399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:28.430944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43536 (local address 127.21.83.61:39897)
0807 07:07:28.431142 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:28.431148 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:28.431166 (+ 18us) server_negotiation.cc:407] Connection header received
0807 07:07:28.431221 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:28.431227 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:28.431299 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:28.431399 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:28.433207 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:28.434114 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:28.435559 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:28.435964 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:28.436052 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:28.440859 (+ 4807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:28.440864 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:28.440867 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:07:28.440926 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:28.443580 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:28.444217 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:28.444223 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:28.444227 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:28.444366 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:28.444727 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:28.444732 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:28.444734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:28.445074 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:28.445329 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:28.445611 (+ 282us) server_negotiation.cc:299] Negotiation successful
0807 07:07:28.445836 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 07:07:28.449306 22502 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7326015bc4ac4e4191d3fd7debacaa4e: Neither blocks nor log segments found. Creating new log.
I20260807 07:07:28.450491 22502 log.cc:824] T 00000000000000000000000000000000 P 7326015bc4ac4e4191d3fd7debacaa4e: Log is configured to *not* fsync() on all Append() calls
I20260807 07:07:28.451296 21836 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 07:07:28.453071 22271 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.83.60" port: 42159 } attempt: 4
I20260807 07:07:28.453743 22502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7326015bc4ac4e4191d3fd7debacaa4e: No bootstrap required, opened a new log
I20260807 07:07:28.455281 22225 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "7161ad4f43d048e489907b04512c1417"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40998
Aug 07 07:07:28 dist-test-slave-ncp9 krb5kdc[22148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086447, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for kudu/127.21.83.60@KRBTEST.COM
I20260807 07:07:28.465039 22502 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7326015bc4ac4e4191d3fd7debacaa4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7161ad4f43d048e489907b04512c1417" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 44033 } } peers { permanent_uuid: "767dbfd3946043968c601e97c4bfd1e2" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39897 } } peers { permanent_uuid: "7326015bc4ac4e4191d3fd7debacaa4e" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 42159 } }
I20260807 07:07:28.465536 22502 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7326015bc4ac4e4191d3fd7debacaa4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7326015bc4ac4e4191d3fd7debacaa4e, State: Initialized, Role: FOLLOWER
I20260807 07:07:28.466117 22502 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7326015bc4ac4e4191d3fd7debacaa4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7161ad4f43d048e489907b04512c1417" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 44033 } } peers { permanent_uuid: "767dbfd3946043968c601e97c4bfd1e2" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39897 } } peers { permanent_uuid: "7326015bc4ac4e4191d3fd7debacaa4e" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 42159 } }
I20260807 07:07:28.467257 22525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7326015bc4ac4e4191d3fd7debacaa4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7161ad4f43d048e489907b04512c1417" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 44033 } } peers { permanent_uuid: "767dbfd3946043968c601e97c4bfd1e2" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39897 } } peers { permanent_uuid: "7326015bc4ac4e4191d3fd7debacaa4e" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 42159 } } }
I20260807 07:07:28.467692 22525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7326015bc4ac4e4191d3fd7debacaa4e [sys.catalog]: This master's current role is: FOLLOWER
I20260807 07:07:28.468642 22502 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7326015bc4ac4e4191d3fd7debacaa4e [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:07:28.469118 22507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:28.454483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54854 (local address 127.21.83.60:42159)
0807 07:07:28.454666 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:28.454671 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:28.454702 (+ 31us) server_negotiation.cc:407] Connection header received
0807 07:07:28.454764 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:28.454768 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:28.454826 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:28.454908 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:28.455771 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:28.456541 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:28.457988 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:28.458527 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:28.458658 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:28.462408 (+ 3750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:28.462414 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:28.462418 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:07:28.462469 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:28.465653 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:28.466613 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:28.466619 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:28.466622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:28.466822 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:28.467325 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:28.467329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:28.467332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:28.467842 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:28.468006 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:28.468475 (+ 469us) server_negotiation.cc:299] Negotiation successful
0807 07:07:28.468736 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 07:07:28.469144 22275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:28.454094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:42159 (local address 127.0.0.1:54854)
0807 07:07:28.454506 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:28.454526 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 07:07:28.454648 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:07:28.454962 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:07:28.454966 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:07:28.455423 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:07:28.455605 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:07:28.455614 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:07:28.456724 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:07:28.456731 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:07:28.457835 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:07:28.457849 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:07:28.458689 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:07:28.458696 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:07:28.458917 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:28.459533 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:07:28.459551 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:07:28.462135 (+ 2584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:07:28.465866 (+ 3731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:07:28.465873 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:07:28.465877 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:07:28.466433 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:07:28.466981 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:07:28.466986 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:07:28.466989 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:07:28.467169 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:07:28.467976 (+ 807us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:07:28.467981 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:07:28.468189 (+ 208us) client_negotiation.cc:769] Sending connection context
0807 07:07:28.468351 (+ 162us) client_negotiation.cc:240] Negotiation successful
0807 07:07:28.468709 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290}
I20260807 07:07:28.474623 22271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7161ad4f43d048e489907b04512c1417: Bootstrap starting.
I20260807 07:07:28.478961 22271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7161ad4f43d048e489907b04512c1417: Neither blocks nor log segments found. Creating new log.
W20260807 07:07:28.480038 22540 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7326015bc4ac4e4191d3fd7debacaa4e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 07:07:28.480187 22540 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 07:07:28.480336 22271 log.cc:824] T 00000000000000000000000000000000 P 7161ad4f43d048e489907b04512c1417: Log is configured to *not* fsync() on all Append() calls
I20260807 07:07:28.483951 22271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7161ad4f43d048e489907b04512c1417: No bootstrap required, opened a new log
I20260807 07:07:28.477656 22403 master_runner.cc:186] Error getting master registration for 127.21.83.62:44033: OK, instance_id { permanent_uuid: "7161ad4f43d048e489907b04512c1417" instance_seqno: 1786086447320234 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 07:07:28.486784 22403 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 07:07:28.492046 22271 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7161ad4f43d048e489907b04512c1417 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7161ad4f43d048e489907b04512c1417" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 44033 } } peers { permanent_uuid: "767dbfd3946043968c601e97c4bfd1e2" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39897 } } peers { permanent_uuid: "7326015bc4ac4e4191d3fd7debacaa4e" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 42159 } }
I20260807 07:07:28.492518 22271 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7161ad4f43d048e489907b04512c1417 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7161ad4f43d048e489907b04512c1417, State: Initialized, Role: FOLLOWER
I20260807 07:07:28.493182 22271 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7161ad4f43d048e489907b04512c1417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7161ad4f43d048e489907b04512c1417" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 44033 } } peers { permanent_uuid: "767dbfd3946043968c601e97c4bfd1e2" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39897 } } peers { permanent_uuid: "7326015bc4ac4e4191d3fd7debacaa4e" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 42159 } }
I20260807 07:07:28.494547 22544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7161ad4f43d048e489907b04512c1417 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7161ad4f43d048e489907b04512c1417" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 44033 } } peers { permanent_uuid: "767dbfd3946043968c601e97c4bfd1e2" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39897 } } peers { permanent_uuid: "7326015bc4ac4e4191d3fd7debacaa4e" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 42159 } } }
I20260807 07:07:28.494968 22544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7161ad4f43d048e489907b04512c1417 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 07:07:28.495846 22271 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7161ad4f43d048e489907b04512c1417 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:07:28.495911 22225 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7161ad4f43d048e489907b04512c1417 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 07:07:28.496104 22225 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7161ad4f43d048e489907b04512c1417 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:07:28.500030 22225 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7161ad4f43d048e489907b04512c1417 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7161ad4f43d048e489907b04512c1417" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 44033 } } peers { permanent_uuid: "767dbfd3946043968c601e97c4bfd1e2" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39897 } } peers { permanent_uuid: "7326015bc4ac4e4191d3fd7debacaa4e" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 42159 } }
I20260807 07:07:28.505028 22463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7161ad4f43d048e489907b04512c1417" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7326015bc4ac4e4191d3fd7debacaa4e"
I20260807 07:07:28.505708 22463 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7326015bc4ac4e4191d3fd7debacaa4e [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:07:28.506904 22225 leader_election.cc:288] T 00000000000000000000000000000000 P 7161ad4f43d048e489907b04512c1417 [CANDIDATE]: Term 1 election: Requested vote from peers 767dbfd3946043968c601e97c4bfd1e2 (127.21.83.61:39897), 7326015bc4ac4e4191d3fd7debacaa4e (127.21.83.60:42159)
I20260807 07:07:28.509500 22463 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7326015bc4ac4e4191d3fd7debacaa4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7161ad4f43d048e489907b04512c1417 in term 1.
W20260807 07:07:28.510015 22555 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7161ad4f43d048e489907b04512c1417: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 07:07:28.510285 22555 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 07:07:28.510447 22192 leader_election.cc:302] T 00000000000000000000000000000000 P 7161ad4f43d048e489907b04512c1417 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7161ad4f43d048e489907b04512c1417, 7326015bc4ac4e4191d3fd7debacaa4e; no voters:
I20260807 07:07:28.511262 22544 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7161ad4f43d048e489907b04512c1417 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:07:28.508167 22168 master_runner.cc:186] Error getting master registration for 127.21.83.61:39897: OK, instance_id { permanent_uuid: "767dbfd3946043968c601e97c4bfd1e2" instance_seqno: 1786086447831944 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 07:07:28.528507 22544 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7161ad4f43d048e489907b04512c1417 [term 1 LEADER]: Becoming Leader. State: Replica: 7161ad4f43d048e489907b04512c1417, State: Running, Role: LEADER
I20260807 07:07:28.529218 22168 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 07:07:28.529575 22544 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7161ad4f43d048e489907b04512c1417 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7161ad4f43d048e489907b04512c1417" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 44033 } } peers { permanent_uuid: "767dbfd3946043968c601e97c4bfd1e2" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39897 } } peers { permanent_uuid: "7326015bc4ac4e4191d3fd7debacaa4e" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 42159 } }
I20260807 07:07:28.540494 22558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7161ad4f43d048e489907b04512c1417 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7161ad4f43d048e489907b04512c1417. Latest consensus state: current_term: 1 leader_uuid: "7161ad4f43d048e489907b04512c1417" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7161ad4f43d048e489907b04512c1417" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 44033 } } peers { permanent_uuid: "767dbfd3946043968c601e97c4bfd1e2" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39897 } } peers { permanent_uuid: "7326015bc4ac4e4191d3fd7debacaa4e" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 42159 } } }
I20260807 07:07:28.540967 22558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7161ad4f43d048e489907b04512c1417 [sys.catalog]: This master's current role is: LEADER
I20260807 07:07:28.546882 22569 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:07:28.549271 22569 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:07:28.558669 22392 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.83.60" port: 42159 } attempt: 5
W20260807 07:07:28.560942 22197 tablet.cc:2369] T 00000000000000000000000000000000 P 7161ad4f43d048e489907b04512c1417: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 07:07:28.563594 22463 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7326015bc4ac4e4191d3fd7debacaa4e [term 1 FOLLOWER]: Refusing update from remote peer 7161ad4f43d048e489907b04512c1417: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:07:28.564998 22558 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7161ad4f43d048e489907b04512c1417 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7326015bc4ac4e4191d3fd7debacaa4e" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 42159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:07:28.590013 22399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:28.555882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43552 (local address 127.21.83.61:39897)
0807 07:07:28.556096 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:28.556102 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:28.570836 (+ 14734us) server_negotiation.cc:407] Connection header received
0807 07:07:28.571166 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:28.571172 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:28.571265 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:28.571341 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:28.573968 (+ 2627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:28.575119 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:28.577218 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:28.577924 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:28.578038 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:28.582175 (+ 4137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:28.582183 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:28.582213 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:07:28.582265 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:28.586196 (+ 3931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:28.587141 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:28.587147 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:28.587152 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:28.587346 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:28.587873 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:28.587878 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:28.587882 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:28.588354 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:28.588622 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:28.589273 (+ 651us) server_negotiation.cc:299] Negotiation successful
0807 07:07:28.589578 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
Aug 07 07:07:28 dist-test-slave-ncp9 krb5kdc[22148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086447, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.61@KRBTEST.COM for kudu/127.21.83.60@KRBTEST.COM
I20260807 07:07:28.609549 22394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:28.559467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:42159 (local address 127.0.0.1:54880)
0807 07:07:28.559837 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:28.559859 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 07:07:28.559992 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:07:28.573978 (+ 13986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:07:28.573984 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:07:28.574551 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:07:28.574780 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:07:28.574794 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:07:28.580881 (+ 6087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:07:28.580887 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:07:28.583875 (+ 2988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:07:28.583895 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:07:28.587980 (+ 4085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:07:28.587988 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:07:28.588288 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:28.589064 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:07:28.589091 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:07:28.592042 (+ 2951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:07:28.601529 (+ 9487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:07:28.601536 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:07:28.601539 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:07:28.602232 (+ 693us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:07:28.606254 (+ 4022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:07:28.606260 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:07:28.606265 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:07:28.606510 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:07:28.608379 (+ 1869us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:07:28.608384 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:07:28.608591 (+ 207us) client_negotiation.cc:769] Sending connection context
0807 07:07:28.608761 (+ 170us) client_negotiation.cc:240] Negotiation successful
0807 07:07:28.609126 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234}
W20260807 07:07:28.610025 22433 tablet.cc:2369] T 00000000000000000000000000000000 P 7326015bc4ac4e4191d3fd7debacaa4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 07:07:28.613229 22525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7326015bc4ac4e4191d3fd7debacaa4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7161ad4f43d048e489907b04512c1417. Latest consensus state: current_term: 1 leader_uuid: "7161ad4f43d048e489907b04512c1417" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7161ad4f43d048e489907b04512c1417" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 44033 } } peers { permanent_uuid: "767dbfd3946043968c601e97c4bfd1e2" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39897 } } peers { permanent_uuid: "7326015bc4ac4e4191d3fd7debacaa4e" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 42159 } } }
I20260807 07:07:28.613576 22525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7326015bc4ac4e4191d3fd7debacaa4e [sys.catalog]: This master's current role is: FOLLOWER
I20260807 07:07:28.616726 22575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:28.560372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54880 (local address 127.21.83.60:42159)
0807 07:07:28.573459 (+ 13087us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:28.573467 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:28.573526 (+ 59us) server_negotiation.cc:407] Connection header received
0807 07:07:28.573619 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:28.573625 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:28.573786 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:28.573956 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:28.575081 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:28.580559 (+ 5478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:28.584404 (+ 3845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:28.587586 (+ 3182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:28.591197 (+ 3611us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:28.592301 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:28.592307 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:28.592312 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:07:28.592352 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:28.598544 (+ 6192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:28.605150 (+ 6606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:28.605156 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:28.605160 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:28.605379 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:28.606690 (+ 1311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:28.606696 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:28.606699 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:28.607104 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:28.610024 (+ 2920us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:28.610176 (+ 152us) server_negotiation.cc:299] Negotiation successful
0807 07:07:28.610432 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12825,"thread_start_us":10347,"threads_started":1}
I20260807 07:07:28.619422 22513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:28.556107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41000 (local address 127.21.83.62:44033)
0807 07:07:28.556566 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:28.556572 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:28.578511 (+ 21939us) server_negotiation.cc:407] Connection header received
0807 07:07:28.581475 (+ 2964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:28.581481 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:28.581585 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:28.581663 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:28.585775 (+ 4112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:28.586939 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:28.595670 (+ 8731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:28.596391 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:28.596541 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:28.611405 (+ 14864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:28.611412 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:28.611417 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:07:28.611479 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:28.615562 (+ 4083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:28.616576 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:28.616582 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:28.616586 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:28.616750 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:28.617493 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:28.617499 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:28.617502 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:28.617894 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:28.618196 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:28.618775 (+ 579us) server_negotiation.cc:299] Negotiation successful
0807 07:07:28.619012 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296}
I20260807 07:07:28.622783 22570 mvcc.cc:205] Tried to move back new op lower bound from 7315810093300973568 to 7315810093183254528. Current Snapshot: MvccSnapshot[applied={T|T < 7315810093300973568}]
I20260807 07:07:28.624269 22392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 767dbfd3946043968c601e97c4bfd1e2: Bootstrap starting.
I20260807 07:07:28.637794 22507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:28.556316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54868 (local address 127.21.83.60:42159)
0807 07:07:28.556736 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:28.556743 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:28.577749 (+ 21006us) server_negotiation.cc:407] Connection header received
0807 07:07:28.579261 (+ 1512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:28.579268 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:28.579388 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:28.579468 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:28.588978 (+ 9510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:28.590055 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:28.602426 (+ 12371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:28.603188 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:28.603329 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:28.611015 (+ 7686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:28.611022 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:28.611027 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:07:28.611067 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:28.616456 (+ 5389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:28.632931 (+ 16475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:28.632938 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:28.632943 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:28.633155 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:28.634553 (+ 1398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:28.634558 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:28.634562 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:28.634979 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:28.635284 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:28.637115 (+ 1831us) server_negotiation.cc:299] Negotiation successful
0807 07:07:28.637400 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277}
I20260807 07:07:28.638384 22392 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 767dbfd3946043968c601e97c4bfd1e2: Neither blocks nor log segments found. Creating new log.
I20260807 07:07:28.640602 22544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7161ad4f43d048e489907b04512c1417 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7161ad4f43d048e489907b04512c1417" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7161ad4f43d048e489907b04512c1417" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 44033 } } peers { permanent_uuid: "767dbfd3946043968c601e97c4bfd1e2" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39897 } } peers { permanent_uuid: "7326015bc4ac4e4191d3fd7debacaa4e" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 42159 } } }
I20260807 07:07:28.640985 22544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7161ad4f43d048e489907b04512c1417 [sys.catalog]: This master's current role is: LEADER
I20260807 07:07:28.642691 22392 log.cc:824] T 00000000000000000000000000000000 P 767dbfd3946043968c601e97c4bfd1e2: Log is configured to *not* fsync() on all Append() calls
I20260807 07:07:28.645675 22525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7326015bc4ac4e4191d3fd7debacaa4e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7161ad4f43d048e489907b04512c1417" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7161ad4f43d048e489907b04512c1417" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 44033 } } peers { permanent_uuid: "767dbfd3946043968c601e97c4bfd1e2" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39897 } } peers { permanent_uuid: "7326015bc4ac4e4191d3fd7debacaa4e" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 42159 } } }
I20260807 07:07:28.645931 22525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7326015bc4ac4e4191d3fd7debacaa4e [sys.catalog]: This master's current role is: FOLLOWER
I20260807 07:07:28.650416 22569 catalog_manager.cc:1347] Generated new cluster ID: 3a0a117c62cf415a9105107a5bdfd70b
I20260807 07:07:28.650533 22569 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:07:28.671447 22569 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:07:28.673441 22569 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:07:28.677302 22392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 767dbfd3946043968c601e97c4bfd1e2: No bootstrap required, opened a new log
I20260807 07:07:28.696871 22569 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7161ad4f43d048e489907b04512c1417: Generated new TSK 0
I20260807 07:07:28.698366 22569 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 07:07:28.698372 22392 raft_consensus.cc:348] T 00000000000000000000000000000000 P 767dbfd3946043968c601e97c4bfd1e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7161ad4f43d048e489907b04512c1417" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 44033 } } peers { permanent_uuid: "767dbfd3946043968c601e97c4bfd1e2" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39897 } } peers { permanent_uuid: "7326015bc4ac4e4191d3fd7debacaa4e" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 42159 } }
I20260807 07:07:28.699167 22392 raft_consensus.cc:729] T 00000000000000000000000000000000 P 767dbfd3946043968c601e97c4bfd1e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 767dbfd3946043968c601e97c4bfd1e2, State: Initialized, Role: FOLLOWER
I20260807 07:07:28.700156 22392 consensus_queue.cc:260] T 00000000000000000000000000000000 P 767dbfd3946043968c601e97c4bfd1e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7161ad4f43d048e489907b04512c1417" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 44033 } } peers { permanent_uuid: "767dbfd3946043968c601e97c4bfd1e2" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39897 } } peers { permanent_uuid: "7326015bc4ac4e4191d3fd7debacaa4e" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 42159 } }
I20260807 07:07:28.703142 22392 sys_catalog.cc:564] T 00000000000000000000000000000000 P 767dbfd3946043968c601e97c4bfd1e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:07:28.701881 22583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 767dbfd3946043968c601e97c4bfd1e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7161ad4f43d048e489907b04512c1417" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 44033 } } peers { permanent_uuid: "767dbfd3946043968c601e97c4bfd1e2" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39897 } } peers { permanent_uuid: "7326015bc4ac4e4191d3fd7debacaa4e" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 42159 } } }
I20260807 07:07:28.705016 22583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 767dbfd3946043968c601e97c4bfd1e2 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 07:07:28.705025 22345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7161ad4f43d048e489907b04512c1417" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "767dbfd3946043968c601e97c4bfd1e2"
I20260807 07:07:28.710353 22344 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 767dbfd3946043968c601e97c4bfd1e2 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:07:28.714244 22344 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 767dbfd3946043968c601e97c4bfd1e2 [term 1 FOLLOWER]: Refusing update from remote peer 7161ad4f43d048e489907b04512c1417: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:07:28.715790 22544 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7161ad4f43d048e489907b04512c1417 [LEADER]: Connected to new peer: Peer: permanent_uuid: "767dbfd3946043968c601e97c4bfd1e2" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:07:28.717806 22345 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 767dbfd3946043968c601e97c4bfd1e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7161ad4f43d048e489907b04512c1417 in term 1.
I20260807 07:07:28.774109 22593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 767dbfd3946043968c601e97c4bfd1e2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7161ad4f43d048e489907b04512c1417" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7161ad4f43d048e489907b04512c1417" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 44033 } } peers { permanent_uuid: "767dbfd3946043968c601e97c4bfd1e2" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39897 } } peers { permanent_uuid: "7326015bc4ac4e4191d3fd7debacaa4e" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 42159 } } }
I20260807 07:07:28.774552 22593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 767dbfd3946043968c601e97c4bfd1e2 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 07:07:28.775197 22583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 767dbfd3946043968c601e97c4bfd1e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7161ad4f43d048e489907b04512c1417. Latest consensus state: current_term: 1 leader_uuid: "7161ad4f43d048e489907b04512c1417" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7161ad4f43d048e489907b04512c1417" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 44033 } } peers { permanent_uuid: "767dbfd3946043968c601e97c4bfd1e2" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39897 } } peers { permanent_uuid: "7326015bc4ac4e4191d3fd7debacaa4e" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 42159 } } }
I20260807 07:07:28.775481 22583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 767dbfd3946043968c601e97c4bfd1e2 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 07:07:28.796049 22605 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260807 07:07:28.824287 21836 mini_kdc.cc:348] Time spent kdestroy: real 0.104s user 0.000s sys 0.032s
W20260807 07:07:28.832497 22507 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:07:28.829553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52074 (local address 127.21.83.60:42159)
0807 07:07:28.829748 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:28.829754 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:28.830500 (+ 746us) server_negotiation.cc:407] Connection header received
0807 07:07:28.830816 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:28.830822 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:28.830913 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:28.830989 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:28.832189 (+ 1200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52074: BlockingRecv error: recv got EOF from 127.0.0.1:52074 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260807 07:07:28.835114 22399 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:07:28.830249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43976 (local address 127.21.83.61:39897)
0807 07:07:28.830436 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:28.830442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:28.833544 (+ 3102us) server_negotiation.cc:407] Connection header received
0807 07:07:28.833809 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:28.833815 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:28.833908 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:28.833975 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:28.834904 (+ 929us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43976: BlockingRecv error: recv got EOF from 127.0.0.1:43976 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260807 07:07:28.838984 22513 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:07:28.836947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33596 (local address 127.21.83.62:44033)
0807 07:07:28.837154 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:28.837160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:28.837370 (+ 210us) server_negotiation.cc:407] Connection header received
0807 07:07:28.837620 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:28.837625 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:28.837700 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:28.837768 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:28.838750 (+ 982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33596: BlockingRecv error: recv got EOF from 127.0.0.1:33596 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260807 07:07:28.877456 22399 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:07:28.875034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43986 (local address 127.21.83.61:39897)
0807 07:07:28.875248 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:28.875255 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:28.875807 (+ 552us) server_negotiation.cc:407] Connection header received
0807 07:07:28.876054 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:28.876058 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:28.876194 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:28.876287 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:28.877150 (+ 863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43986: BlockingRecv error: recv got EOF from 127.0.0.1:43986 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260807 07:07:28.887154 22399 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:07:28.882339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44000 (local address 127.21.83.61:39897)
0807 07:07:28.882531 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:28.882539 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:28.885521 (+ 2982us) server_negotiation.cc:407] Connection header received
0807 07:07:28.885756 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:28.885760 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:28.885834 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:28.885926 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:28.886907 (+ 981us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44000: BlockingRecv error: recv got EOF from 127.0.0.1:44000 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260807 07:07:28.894749 22399 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:07:28.892561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44010 (local address 127.21.83.61:39897)
0807 07:07:28.892770 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:28.892776 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:28.893328 (+ 552us) server_negotiation.cc:407] Connection header received
0807 07:07:28.893523 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:28.893529 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:28.893621 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:28.893712 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:28.894498 (+ 786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44010: BlockingRecv error: recv got EOF from 127.0.0.1:44010 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260807 07:07:28.904492 22399 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:07:28.902307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44014 (local address 127.21.83.61:39897)
0807 07:07:28.902577 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:28.902584 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:28.903114 (+ 530us) server_negotiation.cc:407] Connection header received
0807 07:07:28.903333 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:28.903338 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:28.903413 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:28.903499 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:28.904267 (+ 768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44014: BlockingRecv error: recv got EOF from 127.0.0.1:44014 (error 108)
Metrics: {"server-negotiator.queue_time_us":131}
W20260807 07:07:28.918115 22399 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:07:28.915964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44020 (local address 127.21.83.61:39897)
0807 07:07:28.916200 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:28.916207 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:28.916717 (+ 510us) server_negotiation.cc:407] Connection header received
0807 07:07:28.916932 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:28.916938 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:28.917016 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:28.917105 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:28.917879 (+ 774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44020: BlockingRecv error: recv got EOF from 127.0.0.1:44020 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260807 07:07:28.939847 22399 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:07:28.937681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44030 (local address 127.21.83.61:39897)
0807 07:07:28.937909 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:28.937916 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:28.938431 (+ 515us) server_negotiation.cc:407] Connection header received
0807 07:07:28.938641 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:28.938648 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:28.938749 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:28.938841 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:28.939599 (+ 758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44030: BlockingRecv error: recv got EOF from 127.0.0.1:44030 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260807 07:07:28.983096 22399 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:07:28.980754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44032 (local address 127.21.83.61:39897)
0807 07:07:28.980988 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:28.980996 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:28.981555 (+ 559us) server_negotiation.cc:407] Connection header received
0807 07:07:28.981761 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:28.981765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:28.981842 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:28.981931 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:28.982823 (+ 892us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44032: BlockingRecv error: recv got EOF from 127.0.0.1:44032 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260807 07:07:29.052603 22399 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:07:29.050538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44038 (local address 127.21.83.61:39897)
0807 07:07:29.050771 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:29.050778 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:29.051266 (+ 488us) server_negotiation.cc:407] Connection header received
0807 07:07:29.051464 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:29.051468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:29.051545 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:29.051634 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:29.052389 (+ 755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44038: BlockingRecv error: recv got EOF from 127.0.0.1:44038 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260807 07:07:29.186092 22399 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:07:29.184169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44046 (local address 127.21.83.61:39897)
0807 07:07:29.184381 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:29.184386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:29.184782 (+ 396us) server_negotiation.cc:407] Connection header received
0807 07:07:29.184923 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:29.184929 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:29.185033 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:29.185132 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:29.185869 (+ 737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44046: BlockingRecv error: recv got EOF from 127.0.0.1:44046 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260807 07:07:29.327402 22544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7161ad4f43d048e489907b04512c1417 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7161ad4f43d048e489907b04512c1417" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7161ad4f43d048e489907b04512c1417" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 44033 } } peers { permanent_uuid: "767dbfd3946043968c601e97c4bfd1e2" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 39897 } } peers { permanent_uuid: "7326015bc4ac4e4191d3fd7debacaa4e" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 42159 } } }
I20260807 07:07:29.327759 22544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7161ad4f43d048e489907b04512c1417 [sys.catalog]: This master's current role is: LEADER
W20260807 07:07:29.457237 22399 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:07:29.455234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44062 (local address 127.21.83.61:39897)
0807 07:07:29.455465 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:29.455472 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:29.455590 (+ 118us) server_negotiation.cc:407] Connection header received
0807 07:07:29.455817 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:29.455822 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:29.455935 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:29.456038 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:29.457005 (+ 967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44062: BlockingRecv error: recv got EOF from 127.0.0.1:44062 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
I20260807 07:07:29.482322 22540 catalog_manager.cc:1259] Loaded cluster ID: 3a0a117c62cf415a9105107a5bdfd70b
I20260807 07:07:29.482438 22540 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7326015bc4ac4e4191d3fd7debacaa4e: loading cluster ID for follower catalog manager: success
I20260807 07:07:29.485061 22540 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7326015bc4ac4e4191d3fd7debacaa4e: acquiring CA information for follower catalog manager: success
I20260807 07:07:29.487172 22540 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7326015bc4ac4e4191d3fd7debacaa4e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 07:07:29.802047 22605 catalog_manager.cc:1259] Loaded cluster ID: 3a0a117c62cf415a9105107a5bdfd70b
I20260807 07:07:29.802242 22605 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 767dbfd3946043968c601e97c4bfd1e2: loading cluster ID for follower catalog manager: success
I20260807 07:07:29.806366 22605 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 767dbfd3946043968c601e97c4bfd1e2: acquiring CA information for follower catalog manager: success
I20260807 07:07:29.809361 22605 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 767dbfd3946043968c601e97c4bfd1e2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 07:07:29.983703 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:29.973830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44070 (local address 127.21.83.61:39897)
0807 07:07:29.974732 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:29.974740 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:29.976284 (+ 1544us) server_negotiation.cc:407] Connection header received
0807 07:07:29.976669 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:29.976676 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:29.976958 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:29.977146 (+ 188us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:07:29.977812 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:29.978783 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:29.980851 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:29.981392 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:29.981504 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:29.982219 (+ 715us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:07:29.982927 (+ 708us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:07:29.983067 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:29.983249 (+ 182us) server_negotiation.cc:299] Negotiation successful
0807 07:07:29.983341 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":734,"thread_start_us":495,"threads_started":1}
I20260807 07:07:29.998003 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:29.990540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52084 (local address 127.21.83.60:42159)
0807 07:07:29.991362 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:29.991368 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:29.991757 (+ 389us) server_negotiation.cc:407] Connection header received
0807 07:07:29.991980 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:29.991985 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:29.992155 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:29.992312 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:07:29.992736 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:29.993533 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:29.995412 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:29.995853 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:29.996002 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:29.996632 (+ 630us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:07:29.997221 (+ 589us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:07:29.997357 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:29.997473 (+ 116us) server_negotiation.cc:299] Negotiation successful
0807 07:07:29.997675 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":403,"threads_started":1}
I20260807 07:07:30.025548 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:30.007046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44080 (local address 127.21.83.61:39897)
0807 07:07:30.007308 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:30.007316 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:30.017370 (+ 10054us) server_negotiation.cc:407] Connection header received
0807 07:07:30.017935 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:30.017941 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:30.018046 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:30.018126 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:07:30.018979 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:30.020192 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:30.022735 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:30.023402 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:30.023511 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:30.024560 (+ 1049us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:07:30.024742 (+ 182us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:07:30.024810 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:30.025104 (+ 294us) server_negotiation.cc:299] Negotiation successful
0807 07:07:30.025212 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260807 07:07:30.026135 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:30.006791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52090 (local address 127.21.83.60:42159)
0807 07:07:30.012766 (+ 5975us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:30.012773 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:30.017857 (+ 5084us) server_negotiation.cc:407] Connection header received
0807 07:07:30.017974 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:30.017981 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:30.018113 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:30.018234 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:07:30.019319 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:30.020512 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:30.023066 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:30.023994 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:30.024118 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:30.024915 (+ 797us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:07:30.025067 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:07:30.025162 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:30.025754 (+ 592us) server_negotiation.cc:299] Negotiation successful
0807 07:07:30.025846 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5778}
I20260807 07:07:30.034533 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:30.006480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33602 (local address 127.21.83.62:44033)
0807 07:07:30.017269 (+ 10789us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:30.017276 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:30.028222 (+ 10946us) server_negotiation.cc:407] Connection header received
0807 07:07:30.028570 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:30.028577 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:30.028835 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:30.029022 (+ 187us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:07:30.029646 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:30.030394 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:30.032372 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:30.032828 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:30.033000 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:30.033562 (+ 562us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:07:30.033914 (+ 352us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:07:30.033998 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:30.034154 (+ 156us) server_negotiation.cc:299] Negotiation successful
0807 07:07:30.034237 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10663,"thread_start_us":10561,"threads_started":1}
I20260807 07:07:30.047439 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 22168
I20260807 07:07:30.048643 22263 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:07:30.180012 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 22168
W20260807 07:07:30.209851 22309 connection.cc:535] server connection from 127.0.0.1:43514 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260807 07:07:30.210084 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 22281
I20260807 07:07:30.211385 22385 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:07:30.325403 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 22281
I20260807 07:07:30.350018 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 22403
I20260807 07:07:30.351140 22495 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:07:30.459587 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 22403
2026-08-07T07:07:30Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5228 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 07:07:30 dist-test-slave-ncp9 krb5kdc[22742](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 07:07:30 dist-test-slave-ncp9 krb5kdc[22742](info): set up 2 sockets
Aug 07 07:07:30 dist-test-slave-ncp9 krb5kdc[22742](info): commencing operation
krb5kdc: starting...
W20260807 07:07:31.711033 21836 mini_kdc.cc:121] Time spent starting KDC: real 1.204s user 0.000s sys 0.058s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/krb5kdc/test-user.keytab.
Aug 07 07:07:31 dist-test-slave-ncp9 krb5kdc[22742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T07:07:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T07:07:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
Entry for principal kudu/127.21.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab.
I20260807 07:07:32.076135 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:33025
--webserver_interface=127.21.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:39123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:33025,127.21.83.61:40539,127.21.83.60:45545
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 07:07:32.426600 22763 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 07:07:32.426985 22763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:07:32.427085 22763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:07:32.427217 22763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:07:32.434782 22763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:07:32.434900 22763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:07:32.434984 22763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:07:32.435051 22763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:07:32.435139 22763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:07:32.445365 22763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:39123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:33025,127.21.83.61:40539,127.21.83.60:45545
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.62.keytab
--principal=kudu/127.21.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.62:33025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:07:32.446606 22763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:07:32.447870 22763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:07:32.458947 22768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:07:32.458819 22769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:07:32.459004 22772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:07:32.459980 22763 server_base.cc:1034] running on GCE node
I20260807 07:07:32.460687 22763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:07:32.462030 22763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:07:32.463259 22763 hybrid_clock.cc:648] HybridClock initialized: now 1786086452463187 us; error 43 us; skew 500 ppm
I20260807 07:07:32.463575 22763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:07:32 dist-test-slave-ncp9 krb5kdc[22742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086452, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:07:32.466616 22763 init.cc:377] Logged in from keytab as kudu/127.21.83.62@KRBTEST.COM (short username kudu)
I20260807 07:07:32.467965 22763 webserver.cc:460] Webserver started at http://127.21.83.62:34091/ using document root <none> and password file <none>
I20260807 07:07:32.468597 22763 fs_manager.cc:359] Metadata directory not provided
I20260807 07:07:32.468683 22763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:07:32.468979 22763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:07:32.471948 22763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/minicluster-data/master-0/data/instance:
uuid: "56e35bc7427f4c809d5abc9d8310d981"
format_stamp: "Formatted at 2026-08-07 07:07:32 on dist-test-slave-ncp9"
server_key: "3a1ca94e43d3d51e752513d3093657ca"
server_key_iv: "8c1f62ff17a7411f2dc6968cfcc66ea1"
server_key_version: "encryptionkey@0"
I20260807 07:07:32.472723 22763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/minicluster-data/master-0/wal/instance:
uuid: "56e35bc7427f4c809d5abc9d8310d981"
format_stamp: "Formatted at 2026-08-07 07:07:32 on dist-test-slave-ncp9"
server_key: "3a1ca94e43d3d51e752513d3093657ca"
server_key_iv: "8c1f62ff17a7411f2dc6968cfcc66ea1"
server_key_version: "encryptionkey@0"
I20260807 07:07:32.477973 22763 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260807 07:07:32.481528 22783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:07:32.482928 22763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 07:07:32.483081 22763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/minicluster-data/master-0/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/minicluster-data/master-0/wal
uuid: "56e35bc7427f4c809d5abc9d8310d981"
format_stamp: "Formatted at 2026-08-07 07:07:32 on dist-test-slave-ncp9"
server_key: "3a1ca94e43d3d51e752513d3093657ca"
server_key_iv: "8c1f62ff17a7411f2dc6968cfcc66ea1"
server_key_version: "encryptionkey@0"
I20260807 07:07:32.483269 22763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:07:32.503134 22763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:07:32.506857 22763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:07:32.507073 22763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:07:32.525431 22763 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.62:33025
I20260807 07:07:32.525457 22856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.62:33025 every 8 connection(s)
I20260807 07:07:32.526850 22763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/minicluster-data/master-0/data/info.pb
I20260807 07:07:32.530730 22857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:07:32.533492 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 22763
I20260807 07:07:32.533691 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/minicluster-data/master-0/wal/instance
I20260807 07:07:32.534265 21836 external_mini_cluster.cc:1428] Setting key 1036836469f9ff345f0f39f9231c7de0
I20260807 07:07:32.539142 22857 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.83.62" port: 33025 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 07:07:32 dist-test-slave-ncp9 krb5kdc[22742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086452, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:07:32.562608 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:32.542676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:33025 (local address 127.0.0.1:38834)
0807 07:07:32.543875 (+ 1199us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:32.543919 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 07:07:32.546066 (+ 2147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:07:32.546870 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:07:32.546878 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:07:32.547553 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:07:32.548007 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:07:32.548031 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:07:32.549619 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:07:32.549625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:07:32.551017 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:07:32.551031 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:07:32.552205 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:07:32.552211 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:07:32.552333 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:32.552967 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:07:32.552989 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:07:32.555194 (+ 2205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:07:32.558834 (+ 3640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:07:32.558844 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:07:32.558852 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:07:32.559233 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:07:32.559704 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:07:32.559708 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:07:32.559711 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:07:32.559902 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:07:32.560754 (+ 852us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:07:32.560761 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:07:32.561016 (+ 255us) client_negotiation.cc:769] Sending connection context
0807 07:07:32.561255 (+ 239us) client_negotiation.cc:240] Negotiation successful
0807 07:07:32.561610 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":947,"spinlock_wait_cycles":11648,"thread_start_us":747,"threads_started":1}
I20260807 07:07:32.562645 22859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:32.542685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38834 (local address 127.21.83.62:33025)
0807 07:07:32.543541 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:32.543555 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:32.544046 (+ 491us) server_negotiation.cc:407] Connection header received
0807 07:07:32.546415 (+ 2369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:32.546421 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:32.546646 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:32.546793 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:32.548213 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:32.549453 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:32.551160 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:32.551974 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:32.552172 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:32.555423 (+ 3251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:32.555433 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:32.555445 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:07:32.555478 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:32.558702 (+ 3224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:32.559379 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:32.559387 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:32.559390 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:32.559568 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:32.560041 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:32.560046 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:32.560048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:32.560607 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:32.560795 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:32.561440 (+ 645us) server_negotiation.cc:299] Negotiation successful
0807 07:07:32.561728 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":579,"thread_start_us":338,"threads_started":1}
I20260807 07:07:32.565399 22857 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.83.61" port: 40539 } has no permanent_uuid. Determining permanent_uuid...
I20260807 07:07:32.566725 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:32.566333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.61:40539 (local address 127.0.0.1:52416)
0807 07:07:32.566502 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:32.566609 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.61:40539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260807 07:07:32.567047 22786 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.83.61:40539: connect: Connection refused (error 111)
I20260807 07:07:32.568533 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:32.568211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.61:40539 (local address 127.0.0.1:52418)
0807 07:07:32.568342 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:32.568407 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.61:40539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260807 07:07:32.568738 22857 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.83.61:40539: Network error: Client connection negotiation failed: client connection to 127.21.83.61:40539: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.21.83.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.61.keytab.
Entry for principal kudu/127.21.83.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.61.keytab.
I20260807 07:07:32.611802 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.61:40539
--webserver_interface=127.21.83.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.61.keytab
--principal=kudu/127.21.83.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:39123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:33025,127.21.83.61:40539,127.21.83.60:45545
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 07:07:32.618067 22857 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.83.61" port: 40539 } attempt: 1
I20260807 07:07:32.619251 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:32.618773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.61:40539 (local address 127.0.0.1:52434)
0807 07:07:32.618999 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:32.619107 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.61:40539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260807 07:07:32.620307 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:32.619909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.61:40539 (local address 127.0.0.1:52438)
0807 07:07:32.620063 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:32.620133 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.61:40539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260807 07:07:32.620496 22857 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.83.61:40539: Network error: Client connection negotiation failed: client connection to 127.21.83.61:40539: connect: Connection refused (error 111)
I20260807 07:07:32.688745 22857 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.83.61" port: 40539 } attempt: 2
I20260807 07:07:32.689967 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:32.689445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.61:40539 (local address 127.0.0.1:52446)
0807 07:07:32.689670 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:32.689786 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.61:40539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260807 07:07:32.690995 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:32.690576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.61:40539 (local address 127.0.0.1:52460)
0807 07:07:32.690737 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:32.690839 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.61:40539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260807 07:07:32.691203 22857 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.83.61:40539: Network error: Client connection negotiation failed: client connection to 127.21.83.61:40539: connect: Connection refused (error 111)
I20260807 07:07:32.782480 22857 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.83.61" port: 40539 } attempt: 3
I20260807 07:07:32.783933 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:32.783360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.61:40539 (local address 127.0.0.1:52476)
0807 07:07:32.783617 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:32.783744 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.61:40539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260807 07:07:32.785073 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:32.784586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.61:40539 (local address 127.0.0.1:52484)
0807 07:07:32.784757 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:32.784849 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.61:40539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260807 07:07:32.785315 22857 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.83.61:40539: Network error: Client connection negotiation failed: client connection to 127.21.83.61:40539: connect: Connection refused (error 111)
I20260807 07:07:32.928597 22857 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.83.61" port: 40539 } attempt: 4
I20260807 07:07:32.929927 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:32.929299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.61:40539 (local address 127.0.0.1:52490)
0807 07:07:32.929641 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:32.929755 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.61:40539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180}
I20260807 07:07:32.931035 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:32.930610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.61:40539 (local address 127.0.0.1:52500)
0807 07:07:32.930779 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:32.930884 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.61:40539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260807 07:07:32.931280 22857 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.83.61:40539: Network error: Client connection negotiation failed: client connection to 127.21.83.61:40539: connect: Connection refused (error 111)
W20260807 07:07:32.955154 22868 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 07:07:32.955560 22868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:07:32.955643 22868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:07:32.955726 22868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:07:32.963230 22868 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:07:32.963349 22868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:07:32.963406 22868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:07:32.963471 22868 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:07:32.963534 22868 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:07:32.973778 22868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:39123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:33025,127.21.83.61:40539,127.21.83.60:45545
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.61.keytab
--principal=kudu/127.21.83.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.61:40539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.83.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:07:32.975143 22868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:07:32.976531 22868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:07:32.987851 22876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:07:32.987912 22880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:07:32.987886 22878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:07:32.988711 22868 server_base.cc:1034] running on GCE node
I20260807 07:07:32.989444 22868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:07:32.990645 22868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:07:32.991855 22868 hybrid_clock.cc:648] HybridClock initialized: now 1786086452991781 us; error 46 us; skew 500 ppm
I20260807 07:07:32.992156 22868 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:07:32 dist-test-slave-ncp9 krb5kdc[22742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086452, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:07:32.995078 22868 init.cc:377] Logged in from keytab as kudu/127.21.83.61@KRBTEST.COM (short username kudu)
I20260807 07:07:32.996431 22868 webserver.cc:460] Webserver started at http://127.21.83.61:41263/ using document root <none> and password file <none>
I20260807 07:07:32.997085 22868 fs_manager.cc:359] Metadata directory not provided
I20260807 07:07:32.997175 22868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:07:32.997435 22868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:07:33.000185 22868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/minicluster-data/master-1/data/instance:
uuid: "a67718b839174e6aa7c280484be40a76"
format_stamp: "Formatted at 2026-08-07 07:07:32 on dist-test-slave-ncp9"
server_key: "e940a9d9e4d52a7f6c4249a02d0c4088"
server_key_iv: "fc2099d86d7096a995628b32eb4c34e1"
server_key_version: "encryptionkey@0"
I20260807 07:07:33.000946 22868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/minicluster-data/master-1/wal/instance:
uuid: "a67718b839174e6aa7c280484be40a76"
format_stamp: "Formatted at 2026-08-07 07:07:32 on dist-test-slave-ncp9"
server_key: "e940a9d9e4d52a7f6c4249a02d0c4088"
server_key_iv: "fc2099d86d7096a995628b32eb4c34e1"
server_key_version: "encryptionkey@0"
I20260807 07:07:33.006085 22868 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 07:07:33.009469 22890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:07:33.010826 22868 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 07:07:33.010993 22868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/minicluster-data/master-1/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/minicluster-data/master-1/wal
uuid: "a67718b839174e6aa7c280484be40a76"
format_stamp: "Formatted at 2026-08-07 07:07:32 on dist-test-slave-ncp9"
server_key: "e940a9d9e4d52a7f6c4249a02d0c4088"
server_key_iv: "fc2099d86d7096a995628b32eb4c34e1"
server_key_version: "encryptionkey@0"
I20260807 07:07:33.011193 22868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:07:33.043176 22868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:07:33.047940 22868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:07:33.048236 22868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:07:33.067355 22868 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.61:40539
I20260807 07:07:33.067397 22965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.61:40539 every 8 connection(s)
I20260807 07:07:33.068800 22868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/minicluster-data/master-1/data/info.pb
I20260807 07:07:33.072750 22966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:07:33.077945 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 22868
I20260807 07:07:33.078155 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/minicluster-data/master-1/wal/instance
I20260807 07:07:33.078723 21836 external_mini_cluster.cc:1428] Setting key c36a83f3ceff00554668638a07266aa2
I20260807 07:07:33.080801 22966 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.83.62" port: 33025 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 07:07:33 dist-test-slave-ncp9 krb5kdc[22742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086452, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.61@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:07:33.105249 22971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:33.084387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38836 (local address 127.21.83.62:33025)
0807 07:07:33.084990 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:33.084997 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:33.086001 (+ 1004us) server_negotiation.cc:407] Connection header received
0807 07:07:33.088541 (+ 2540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:33.088545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:33.088635 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:33.088740 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:33.090994 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:33.091813 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:33.093990 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:33.094431 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:33.094538 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:33.098669 (+ 4131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:33.098675 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:33.098680 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:07:33.098736 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:33.101955 (+ 3219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:33.102761 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:33.102768 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:33.102771 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:33.102951 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:33.103349 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:33.103354 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:33.103356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:33.103724 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:33.103876 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:33.104656 (+ 780us) server_negotiation.cc:299] Negotiation successful
0807 07:07:33.104873 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":290,"threads_started":1}
I20260807 07:07:33.106164 22972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:33.084927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:33025 (local address 127.0.0.1:38836)
0807 07:07:33.085817 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:33.085876 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0807 07:07:33.088206 (+ 2330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:07:33.089063 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:07:33.089075 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:07:33.089864 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:07:33.090695 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:07:33.090722 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:07:33.092000 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:07:33.092005 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:07:33.093813 (+ 1808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:07:33.093830 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:07:33.094669 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:07:33.094675 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:07:33.094955 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:33.095737 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:07:33.095766 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:07:33.098371 (+ 2605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:07:33.102094 (+ 3723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:07:33.102102 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:07:33.102117 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:07:33.102598 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:07:33.103052 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:07:33.103055 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:07:33.103057 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:07:33.103237 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:07:33.103883 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:07:33.103890 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:07:33.104252 (+ 362us) client_negotiation.cc:769] Sending connection context
0807 07:07:33.104511 (+ 259us) client_negotiation.cc:240] Negotiation successful
0807 07:07:33.104966 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":347,"threads_started":1}
I20260807 07:07:33.107846 22966 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.83.61" port: 40539 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.21.83.60@KRBTEST.COM; defaulting to no policy
Aug 07 07:07:33 dist-test-slave-ncp9 krb5kdc[22742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086452, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.61@KRBTEST.COM for kudu/127.21.83.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.83.60@KRBTEST.COM" created.
I20260807 07:07:33.122275 22972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:33.108716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.61:40539 (local address 127.0.0.1:52506)
0807 07:07:33.108923 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:33.108943 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 07:07:33.109047 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:07:33.110011 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:07:33.110015 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:07:33.110436 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:07:33.110579 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:07:33.110587 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:07:33.111917 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:07:33.111922 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:07:33.112750 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:07:33.112761 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:07:33.113592 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:07:33.113598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:07:33.113756 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:33.114312 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:07:33.114336 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:07:33.116751 (+ 2415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:07:33.119981 (+ 3230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:07:33.119985 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:07:33.119988 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:07:33.120355 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:07:33.120731 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:07:33.120734 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:07:33.120737 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:07:33.120861 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:07:33.121554 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:07:33.121557 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:07:33.121682 (+ 125us) client_negotiation.cc:769] Sending connection context
0807 07:07:33.121796 (+ 114us) client_negotiation.cc:240] Negotiation successful
0807 07:07:33.121993 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260807 07:07:33.122607 22974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:33.109009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52506 (local address 127.21.83.61:40539)
0807 07:07:33.109628 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:33.109634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:33.109659 (+ 25us) server_negotiation.cc:407] Connection header received
0807 07:07:33.109700 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:33.109705 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:33.109869 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:33.110003 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:33.110736 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:33.111786 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:33.112875 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:33.113434 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:33.113545 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:33.117000 (+ 3455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:33.117011 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:33.117025 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:07:33.117054 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:33.119852 (+ 2798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:33.120480 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:33.120488 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:33.120492 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:33.120634 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:33.120987 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:33.120991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:33.120994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:33.121432 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:33.121650 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:33.121934 (+ 284us) server_negotiation.cc:299] Negotiation successful
0807 07:07:33.122205 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":345,"threads_started":1}
I20260807 07:07:33.125037 22966 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.83.60" port: 45545 } has no permanent_uuid. Determining permanent_uuid...
I20260807 07:07:33.126191 22972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:33.125736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:45545 (local address 127.0.0.1:39620)
0807 07:07:33.125914 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:33.126057 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:45545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260807 07:07:33.126452 22893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.83.60:45545: connect: Connection refused (error 111)
I20260807 07:07:33.127924 22972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:33.127618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:45545 (local address 127.0.0.1:39626)
0807 07:07:33.127752 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:33.127822 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:45545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260807 07:07:33.128124 22966 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.83.60:45545: Network error: Client connection negotiation failed: client connection to 127.21.83.60:45545: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.83.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.60.keytab.
Entry for principal kudu/127.21.83.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.60.keytab.
I20260807 07:07:33.157760 21836 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
/tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.60:45545
--webserver_interface=127.21.83.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.60.keytab
--principal=kudu/127.21.83.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.83.20:39123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.83.62:33025,127.21.83.61:40539,127.21.83.60:45545
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 07:07:33.177430 22966 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.83.60" port: 45545 } attempt: 1
I20260807 07:07:33.178716 22972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:33.178202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:45545 (local address 127.0.0.1:39632)
0807 07:07:33.178418 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:33.178537 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:45545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260807 07:07:33.179726 22972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:33.179331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:45545 (local address 127.0.0.1:39648)
0807 07:07:33.179508 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:33.179603 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:45545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260807 07:07:33.179931 22966 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.83.60:45545: Network error: Client connection negotiation failed: client connection to 127.21.83.60:45545: connect: Connection refused (error 111)
I20260807 07:07:33.230526 22857 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.83.61" port: 40539 } attempt: 5
Aug 07 07:07:33 dist-test-slave-ncp9 krb5kdc[22742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086452, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for kudu/127.21.83.61@KRBTEST.COM
I20260807 07:07:33.245463 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:33.231263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.61:40539 (local address 127.0.0.1:52516)
0807 07:07:33.231507 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:33.231532 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 07:07:33.231689 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:07:33.232083 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:07:33.232087 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:07:33.232513 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:07:33.232706 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:07:33.232716 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:07:33.234203 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:07:33.234208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:07:33.235156 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:07:33.235169 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:07:33.236214 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:07:33.236220 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:07:33.236387 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:33.237083 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:07:33.237109 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:07:33.239393 (+ 2284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:07:33.243114 (+ 3721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:07:33.243119 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:07:33.243122 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:07:33.243507 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:07:33.243894 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:07:33.243897 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:07:33.243900 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:07:33.244028 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:07:33.244630 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:07:33.244634 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:07:33.244775 (+ 141us) client_negotiation.cc:769] Sending connection context
0807 07:07:33.244972 (+ 197us) client_negotiation.cc:240] Negotiation successful
0807 07:07:33.245200 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260807 07:07:33.245647 22974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:33.231525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52516 (local address 127.21.83.61:40539)
0807 07:07:33.231740 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:33.231747 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:33.231770 (+ 23us) server_negotiation.cc:407] Connection header received
0807 07:07:33.231835 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:33.231840 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:33.231938 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:33.232051 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:33.232915 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:33.234021 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:33.235331 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:33.236030 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:33.236188 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:33.239618 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:33.239626 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:33.239631 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:07:33.239673 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:33.242975 (+ 3302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:33.243621 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:33.243626 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:33.243630 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:33.243796 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:33.244133 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:33.244137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:33.244140 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:33.244514 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:33.244678 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:33.245082 (+ 404us) server_negotiation.cc:299] Negotiation successful
0807 07:07:33.245328 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 07:07:33.246662 22857 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.83.60" port: 45545 } has no permanent_uuid. Determining permanent_uuid...
I20260807 07:07:33.247917 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:33.247397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:45545 (local address 127.0.0.1:39650)
0807 07:07:33.247604 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:33.247715 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:45545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260807 07:07:33.248132 22966 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.83.60" port: 45545 } attempt: 2
I20260807 07:07:33.248987 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:33.248537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:45545 (local address 127.0.0.1:39652)
0807 07:07:33.248703 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:33.248818 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:45545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260807 07:07:33.249096 22972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:33.248661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:45545 (local address 127.0.0.1:39666)
0807 07:07:33.248820 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:33.248928 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:45545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260807 07:07:33.249406 22857 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.83.60:45545: Network error: Client connection negotiation failed: client connection to 127.21.83.60:45545: connect: Connection refused (error 111)
I20260807 07:07:33.250021 22972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:33.249696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:45545 (local address 127.0.0.1:39678)
0807 07:07:33.249841 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:33.249912 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:45545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260807 07:07:33.250255 22966 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.83.60:45545: Network error: Client connection negotiation failed: client connection to 127.21.83.60:45545: connect: Connection refused (error 111)
I20260807 07:07:33.300652 22857 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.83.60" port: 45545 } attempt: 1
I20260807 07:07:33.301901 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:33.301331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:45545 (local address 127.0.0.1:39682)
0807 07:07:33.301562 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:33.301683 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:45545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260807 07:07:33.302973 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:33.302555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:45545 (local address 127.0.0.1:39694)
0807 07:07:33.302733 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:33.302830 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:45545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260807 07:07:33.303205 22857 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.83.60:45545: Network error: Client connection negotiation failed: client connection to 127.21.83.60:45545: connect: Connection refused (error 111)
I20260807 07:07:33.341497 22966 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.83.60" port: 45545 } attempt: 3
I20260807 07:07:33.342867 22972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:33.342322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:45545 (local address 127.0.0.1:39696)
0807 07:07:33.342574 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:33.342689 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:45545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260807 07:07:33.344092 22972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:33.343565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:45545 (local address 127.0.0.1:39712)
0807 07:07:33.343853 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:33.343945 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:45545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184}
W20260807 07:07:33.344328 22966 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.83.60:45545: Network error: Client connection negotiation failed: client connection to 127.21.83.60:45545: connect: Connection refused (error 111)
I20260807 07:07:33.371476 22857 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.83.60" port: 45545 } attempt: 2
I20260807 07:07:33.372987 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:33.372275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:45545 (local address 127.0.0.1:39724)
0807 07:07:33.372559 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:33.372705 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:45545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260807 07:07:33.374269 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:33.373837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:45545 (local address 127.0.0.1:39732)
0807 07:07:33.374020 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:33.374130 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:45545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260807 07:07:33.374550 22857 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.83.60:45545: Network error: Client connection negotiation failed: client connection to 127.21.83.60:45545: connect: Connection refused (error 111)
I20260807 07:07:33.480822 22857 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.83.60" port: 45545 } attempt: 3
I20260807 07:07:33.482245 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:33.481653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:45545 (local address 127.0.0.1:39740)
0807 07:07:33.481897 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:33.482014 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:45545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260807 07:07:33.483518 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:33.483003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:45545 (local address 127.0.0.1:39750)
0807 07:07:33.483273 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:33.483378 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:45545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20260807 07:07:33.483789 22857 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.83.60:45545: Network error: Client connection negotiation failed: client connection to 127.21.83.60:45545: connect: Connection refused (error 111)
I20260807 07:07:33.487524 22966 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.83.60" port: 45545 } attempt: 4
I20260807 07:07:33.488534 22972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:33.488113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:45545 (local address 127.0.0.1:39762)
0807 07:07:33.488315 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:33.488395 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:45545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260807 07:07:33.489449 22972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:33.489149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:45545 (local address 127.0.0.1:39764)
0807 07:07:33.489284 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:33.489349 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.83.60:45545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260807 07:07:33.489652 22966 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.83.60:45545: Network error: Client connection negotiation failed: client connection to 127.21.83.60:45545: connect: Connection refused (error 111)
W20260807 07:07:33.517376 22977 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 07:07:33.517699 22977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 07:07:33.517800 22977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 07:07:33.517910 22977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 07:07:33.525269 22977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 07:07:33.525375 22977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 07:07:33.525463 22977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 07:07:33.525529 22977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 07:07:33.525626 22977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 07:07:33.535701 22977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.83.20:39123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.83.62:33025,127.21.83.61:40539,127.21.83.60:45545
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/krb5kdc/kudu_127.21.83.60.keytab
--principal=kudu/127.21.83.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.83.60:45545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.83.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 07:07:33.537043 22977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:07:33.538378 22977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:07:33.549777 22985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:07:33.549769 22988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:07:33.549768 22991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:07:33.550561 22977 server_base.cc:1034] running on GCE node
I20260807 07:07:33.551308 22977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 07:07:33.552608 22977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 07:07:33.553872 22977 hybrid_clock.cc:648] HybridClock initialized: now 1786086453553791 us; error 35 us; skew 500 ppm
I20260807 07:07:33.554234 22977 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 07:07:33 dist-test-slave-ncp9 krb5kdc[22742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086453, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 07:07:33.557415 22977 init.cc:377] Logged in from keytab as kudu/127.21.83.60@KRBTEST.COM (short username kudu)
I20260807 07:07:33.559008 22977 webserver.cc:460] Webserver started at http://127.21.83.60:45435/ using document root <none> and password file <none>
I20260807 07:07:33.559723 22977 fs_manager.cc:359] Metadata directory not provided
I20260807 07:07:33.559824 22977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:07:33.560128 22977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:07:33.563537 22977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/minicluster-data/master-2/data/instance:
uuid: "0fa321ed1910400eb592b1a9c1348698"
format_stamp: "Formatted at 2026-08-07 07:07:33 on dist-test-slave-ncp9"
server_key: "1e59bdc0149faf5b6a77ded2ab6df51b"
server_key_iv: "b9b7671193215e896d6365752476335b"
server_key_version: "encryptionkey@0"
I20260807 07:07:33.564473 22977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/minicluster-data/master-2/wal/instance:
uuid: "0fa321ed1910400eb592b1a9c1348698"
format_stamp: "Formatted at 2026-08-07 07:07:33 on dist-test-slave-ncp9"
server_key: "1e59bdc0149faf5b6a77ded2ab6df51b"
server_key_iv: "b9b7671193215e896d6365752476335b"
server_key_version: "encryptionkey@0"
I20260807 07:07:33.570310 22977 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260807 07:07:33.573890 23000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:07:33.575225 22977 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 07:07:33.575390 22977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/minicluster-data/master-2/data,/tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/minicluster-data/master-2/wal
uuid: "0fa321ed1910400eb592b1a9c1348698"
format_stamp: "Formatted at 2026-08-07 07:07:33 on dist-test-slave-ncp9"
server_key: "1e59bdc0149faf5b6a77ded2ab6df51b"
server_key_iv: "b9b7671193215e896d6365752476335b"
server_key_version: "encryptionkey@0"
I20260807 07:07:33.575616 22977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:07:33.601114 22977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:07:33.604719 22977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:07:33.604971 22977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:07:33.622256 22977 rpc_server.cc:307] RPC server started. Bound to: 127.21.83.60:45545
I20260807 07:07:33.622274 23083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.83.60:45545 every 8 connection(s)
I20260807 07:07:33.623508 22977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/minicluster-data/master-2/data/info.pb
I20260807 07:07:33.626055 21836 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu as pid 22977
I20260807 07:07:33.626259 21836 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyUa4Vb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786086078080656-21836-0/minicluster-data/master-2/wal/instance
I20260807 07:07:33.626847 21836 external_mini_cluster.cc:1428] Setting key 347397ea3eb58571405df4f88147df31
I20260807 07:07:33.627748 23084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:07:33.637274 23084 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.83.62" port: 33025 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 07:07:33 dist-test-slave-ncp9 krb5kdc[22742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.60@KRBTEST.COM
Aug 07 07:07:33 dist-test-slave-ncp9 krb5kdc[22742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086453, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.60@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
I20260807 07:07:33.649827 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:33.628655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39770 (local address 127.21.83.60:45545)
0807 07:07:33.629408 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:33.629420 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:33.629458 (+ 38us) server_negotiation.cc:407] Connection header received
0807 07:07:33.630940 (+ 1482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:33.630960 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:33.631325 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:33.631649 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:33.633585 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:33.634908 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:33.636725 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:33.637411 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:33.637542 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:33.643023 (+ 5481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:33.643030 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:33.643038 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:07:33.643071 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:33.645868 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:33.646631 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:33.646636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:33.646642 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:33.646801 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:33.647201 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:33.647204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:33.647206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:33.647704 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:33.647997 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:33.648340 (+ 343us) server_negotiation.cc:299] Negotiation successful
0807 07:07:33.648761 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":351,"threads_started":1}
I20260807 07:07:33.655022 23088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:33.640170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38840 (local address 127.21.83.62:33025)
0807 07:07:33.640680 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:33.640687 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:33.641102 (+ 415us) server_negotiation.cc:407] Connection header received
0807 07:07:33.641302 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:33.641309 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:33.641416 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:33.641551 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:33.642811 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:33.643498 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:33.645085 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:33.645511 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:33.645625 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:33.649191 (+ 3566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:33.649196 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:33.649200 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:07:33.649234 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:33.652140 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:33.652884 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:33.652913 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:33.652916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:33.653100 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:33.653617 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:33.653622 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:33.653624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:33.653926 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:33.654061 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:33.654534 (+ 473us) server_negotiation.cc:299] Negotiation successful
0807 07:07:33.654720 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":224,"threads_started":1}
I20260807 07:07:33.655212 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:33.640215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.62:33025 (local address 127.0.0.1:38840)
0807 07:07:33.640957 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:33.641004 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 07:07:33.641152 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:07:33.641583 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:07:33.641588 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:07:33.642289 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:07:33.642633 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:07:33.642647 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:07:33.643666 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:07:33.643671 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:07:33.644938 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:07:33.644951 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:07:33.645714 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:07:33.645720 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:07:33.645865 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:33.646407 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:07:33.646430 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:07:33.649003 (+ 2573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:07:33.652271 (+ 3268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:07:33.652282 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:07:33.652289 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:07:33.652731 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:07:33.653229 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:07:33.653233 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:07:33.653237 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:07:33.653424 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:07:33.654046 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:07:33.654054 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:07:33.654255 (+ 201us) client_negotiation.cc:769] Sending connection context
0807 07:07:33.654459 (+ 204us) client_negotiation.cc:240] Negotiation successful
0807 07:07:33.654796 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":355,"threads_started":1}
I20260807 07:07:33.657009 23084 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.83.61" port: 40539 } has no permanent_uuid. Determining permanent_uuid...
I20260807 07:07:33.660987 22857 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.83.60" port: 45545 } attempt: 4
Aug 07 07:07:33 dist-test-slave-ncp9 krb5kdc[22742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.62@KRBTEST.COM
Aug 07 07:07:33 dist-test-slave-ncp9 krb5kdc[22742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086453, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.60@KRBTEST.COM for kudu/127.21.83.61@KRBTEST.COM
Aug 07 07:07:33 dist-test-slave-ncp9 krb5kdc[22742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086452, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.62@KRBTEST.COM for kudu/127.21.83.60@KRBTEST.COM
I20260807 07:07:33.670760 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:33.652781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38846 (local address 127.21.83.62:33025)
0807 07:07:33.653462 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:33.653467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:33.653491 (+ 24us) server_negotiation.cc:407] Connection header received
0807 07:07:33.653539 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:33.653544 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:33.653611 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:33.653696 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:33.655231 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:33.656407 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:33.657728 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:33.658332 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:33.658452 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:33.663669 (+ 5217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:33.663676 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:33.663681 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:07:33.663722 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:33.667255 (+ 3533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:33.668095 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:33.668101 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:33.668105 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:33.668303 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:33.668667 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:33.668672 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:33.668676 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:33.669020 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:33.669291 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:33.670175 (+ 884us) server_negotiation.cc:299] Negotiation successful
0807 07:07:33.670413 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"spinlock_wait_cycles":1280,"thread_start_us":313,"threads_started":1}
I20260807 07:07:33.673025 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:33.657680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.61:40539 (local address 127.0.0.1:52530)
0807 07:07:33.657959 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:33.657989 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0807 07:07:33.658104 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:07:33.658391 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:07:33.658396 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:07:33.658825 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:07:33.659009 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:07:33.659021 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:07:33.660321 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:07:33.660325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:07:33.661402 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:07:33.661415 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:07:33.662389 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:07:33.662395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:07:33.662647 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:33.663250 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:07:33.663267 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:07:33.665860 (+ 2593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:07:33.669495 (+ 3635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:07:33.669500 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:07:33.669503 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:07:33.670010 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:07:33.670461 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:07:33.670466 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:07:33.670470 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:07:33.670645 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:07:33.672017 (+ 1372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:07:33.672023 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:07:33.672220 (+ 197us) client_negotiation.cc:769] Sending connection context
0807 07:07:33.672379 (+ 159us) client_negotiation.cc:240] Negotiation successful
0807 07:07:33.672669 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168}
I20260807 07:07:33.674041 22974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:33.657797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52530 (local address 127.21.83.61:40539)
0807 07:07:33.657961 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:33.657965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:33.658049 (+ 84us) server_negotiation.cc:407] Connection header received
0807 07:07:33.658199 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:33.658203 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:33.658265 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:33.658341 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:33.659165 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:33.660173 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:33.661564 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:33.662220 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:33.662363 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:33.666084 (+ 3721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:33.666091 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:33.666096 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:07:33.666139 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:33.669359 (+ 3220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:33.670156 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:33.670161 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:33.670164 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:33.670332 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:33.670769 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:33.670774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:33.670777 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:33.671858 (+ 1081us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:33.672017 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:33.673422 (+ 1405us) server_negotiation.cc:299] Negotiation successful
0807 07:07:33.673643 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 07:07:33.675573 23084 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.83.60" port: 45545 } has no permanent_uuid. Determining permanent_uuid...
I20260807 07:07:33.676985 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:33.661628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:45545 (local address 127.0.0.1:39772)
0807 07:07:33.661804 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:33.661823 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 07:07:33.661926 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:07:33.663257 (+ 1331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:07:33.663263 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:07:33.663723 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:07:33.663906 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:07:33.663915 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:07:33.665079 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:07:33.665085 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:07:33.666422 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:07:33.666438 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:07:33.667377 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:07:33.667384 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:07:33.667591 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:33.668104 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:07:33.668127 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:07:33.670750 (+ 2623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:07:33.674424 (+ 3674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:07:33.674429 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:07:33.674432 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:07:33.674908 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:07:33.675308 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:07:33.675312 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:07:33.675315 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:07:33.675472 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:07:33.676019 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:07:33.676025 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:07:33.676184 (+ 159us) client_negotiation.cc:769] Sending connection context
0807 07:07:33.676321 (+ 137us) client_negotiation.cc:240] Negotiation successful
0807 07:07:33.676594 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260807 07:07:33.677830 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:33.662449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39772 (local address 127.21.83.60:45545)
0807 07:07:33.662643 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:33.662649 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:33.662668 (+ 19us) server_negotiation.cc:407] Connection header received
0807 07:07:33.662715 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:33.662719 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:33.662786 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:33.662896 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:33.664058 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:33.664948 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:33.666593 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:33.667213 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:33.667343 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:33.671083 (+ 3740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:33.671087 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:33.671090 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:07:33.671118 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:33.674273 (+ 3155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:33.675004 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:33.675009 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:33.675013 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:33.675175 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:33.675575 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:33.675580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:33.675583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:33.675891 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:33.676018 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:33.677247 (+ 1229us) server_negotiation.cc:299] Negotiation successful
0807 07:07:33.677492 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Aug 07 07:07:33 dist-test-slave-ncp9 krb5kdc[22742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.83.61@KRBTEST.COM
I20260807 07:07:33.683331 22857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56e35bc7427f4c809d5abc9d8310d981: Bootstrap starting.
Aug 07 07:07:33 dist-test-slave-ncp9 krb5kdc[22742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086453, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.60@KRBTEST.COM for kudu/127.21.83.60@KRBTEST.COM
I20260807 07:07:33.687136 22857 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56e35bc7427f4c809d5abc9d8310d981: Neither blocks nor log segments found. Creating new log.
I20260807 07:07:33.688334 22857 log.cc:824] T 00000000000000000000000000000000 P 56e35bc7427f4c809d5abc9d8310d981: Log is configured to *not* fsync() on all Append() calls
I20260807 07:07:33.690368 22974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:33.672532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52542 (local address 127.21.83.61:40539)
0807 07:07:33.674366 (+ 1834us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:33.674371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:33.674387 (+ 16us) server_negotiation.cc:407] Connection header received
0807 07:07:33.674430 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:33.674434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:33.674501 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:33.674594 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:33.676136 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:33.677165 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:33.678783 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:33.679282 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:33.679389 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:33.684058 (+ 4669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:33.684063 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:33.684067 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:07:33.684102 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:33.687472 (+ 3370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:33.688234 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:33.688240 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:33.688243 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:33.688441 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:33.688957 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:33.688973 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:33.688977 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:33.689274 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:33.689514 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:33.689816 (+ 302us) server_negotiation.cc:299] Negotiation successful
0807 07:07:33.690032 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1735,"thread_start_us":2648,"threads_started":1}
I20260807 07:07:33.690644 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:33.676281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:45545 (local address 127.0.0.1:39784)
0807 07:07:33.676432 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:33.676449 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 07:07:33.676547 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:07:33.677559 (+ 1012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:07:33.677564 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:07:33.678018 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:07:33.678187 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:07:33.678195 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:07:33.679762 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:07:33.679767 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:07:33.681033 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:07:33.681048 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:07:33.681954 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:07:33.681960 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:07:33.682159 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:33.683131 (+ 972us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:07:33.683153 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:07:33.685287 (+ 2134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:07:33.688281 (+ 2994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:07:33.688285 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:07:33.688289 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:07:33.688675 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:07:33.689049 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:07:33.689054 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:07:33.689057 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:07:33.689203 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:07:33.689777 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:07:33.689782 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:07:33.689917 (+ 135us) client_negotiation.cc:769] Sending connection context
0807 07:07:33.690033 (+ 116us) client_negotiation.cc:240] Negotiation successful
0807 07:07:33.690320 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260807 07:07:33.691217 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:33.676540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39784 (local address 127.21.83.60:45545)
0807 07:07:33.677313 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:33.677317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:33.677334 (+ 17us) server_negotiation.cc:407] Connection header received
0807 07:07:33.677372 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:33.677375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:33.677427 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:33.677520 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:33.678345 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:33.679652 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:33.681191 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:33.681789 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:33.681907 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:33.685467 (+ 3560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:33.685479 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:33.685483 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:07:33.685509 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:33.688193 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:33.688771 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:33.688776 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:33.688779 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:33.688935 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:33.689334 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:33.689338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:33.689341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:33.689642 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:33.689817 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:33.690142 (+ 325us) server_negotiation.cc:299] Negotiation successful
0807 07:07:33.690967 (+ 825us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":130,"server-negotiator.queue_time_us":502,"thread_start_us":356,"threads_started":1}
I20260807 07:07:33.692147 22857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56e35bc7427f4c809d5abc9d8310d981: No bootstrap required, opened a new log
I20260807 07:07:33.695088 21836 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 07:07:33.695911 23084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fa321ed1910400eb592b1a9c1348698: Bootstrap starting.
I20260807 07:07:33.698040 22820 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "56e35bc7427f4c809d5abc9d8310d981"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38846
I20260807 07:07:33.699414 23084 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fa321ed1910400eb592b1a9c1348698: Neither blocks nor log segments found. Creating new log.
I20260807 07:07:33.700129 22857 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56e35bc7427f4c809d5abc9d8310d981 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e35bc7427f4c809d5abc9d8310d981" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 33025 } } peers { permanent_uuid: "a67718b839174e6aa7c280484be40a76" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 40539 } } peers { permanent_uuid: "0fa321ed1910400eb592b1a9c1348698" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 45545 } }
I20260807 07:07:33.700414 22857 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56e35bc7427f4c809d5abc9d8310d981 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e35bc7427f4c809d5abc9d8310d981, State: Initialized, Role: FOLLOWER
I20260807 07:07:33.700593 23084 log.cc:824] T 00000000000000000000000000000000 P 0fa321ed1910400eb592b1a9c1348698: Log is configured to *not* fsync() on all Append() calls
I20260807 07:07:33.700847 22857 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56e35bc7427f4c809d5abc9d8310d981 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e35bc7427f4c809d5abc9d8310d981" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 33025 } } peers { permanent_uuid: "a67718b839174e6aa7c280484be40a76" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 40539 } } peers { permanent_uuid: "0fa321ed1910400eb592b1a9c1348698" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 45545 } }
I20260807 07:07:33.701766 23101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56e35bc7427f4c809d5abc9d8310d981 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e35bc7427f4c809d5abc9d8310d981" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 33025 } } peers { permanent_uuid: "a67718b839174e6aa7c280484be40a76" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 40539 } } peers { permanent_uuid: "0fa321ed1910400eb592b1a9c1348698" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 45545 } } }
I20260807 07:07:33.702184 23101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56e35bc7427f4c809d5abc9d8310d981 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 07:07:33.703011 22857 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56e35bc7427f4c809d5abc9d8310d981 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:07:33.703060 22820 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56e35bc7427f4c809d5abc9d8310d981 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 07:07:33.703235 22820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56e35bc7427f4c809d5abc9d8310d981 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:07:33.703517 23084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fa321ed1910400eb592b1a9c1348698: No bootstrap required, opened a new log
I20260807 07:07:33.706946 22820 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56e35bc7427f4c809d5abc9d8310d981 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e35bc7427f4c809d5abc9d8310d981" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 33025 } } peers { permanent_uuid: "a67718b839174e6aa7c280484be40a76" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 40539 } } peers { permanent_uuid: "0fa321ed1910400eb592b1a9c1348698" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 45545 } }
I20260807 07:07:33.709424 22820 leader_election.cc:288] T 00000000000000000000000000000000 P 56e35bc7427f4c809d5abc9d8310d981 [CANDIDATE]: Term 1 election: Requested vote from peers a67718b839174e6aa7c280484be40a76 (127.21.83.61:40539), 0fa321ed1910400eb592b1a9c1348698 (127.21.83.60:45545)
I20260807 07:07:33.712060 23084 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fa321ed1910400eb592b1a9c1348698 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e35bc7427f4c809d5abc9d8310d981" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 33025 } } peers { permanent_uuid: "a67718b839174e6aa7c280484be40a76" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 40539 } } peers { permanent_uuid: "0fa321ed1910400eb592b1a9c1348698" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 45545 } }
I20260807 07:07:33.712464 23084 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fa321ed1910400eb592b1a9c1348698 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fa321ed1910400eb592b1a9c1348698, State: Initialized, Role: FOLLOWER
I20260807 07:07:33.712982 23084 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fa321ed1910400eb592b1a9c1348698 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e35bc7427f4c809d5abc9d8310d981" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 33025 } } peers { permanent_uuid: "a67718b839174e6aa7c280484be40a76" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 40539 } } peers { permanent_uuid: "0fa321ed1910400eb592b1a9c1348698" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 45545 } }
I20260807 07:07:33.714403 23114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fa321ed1910400eb592b1a9c1348698 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e35bc7427f4c809d5abc9d8310d981" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 33025 } } peers { permanent_uuid: "a67718b839174e6aa7c280484be40a76" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 40539 } } peers { permanent_uuid: "0fa321ed1910400eb592b1a9c1348698" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 45545 } } }
I20260807 07:07:33.715482 23114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fa321ed1910400eb592b1a9c1348698 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 07:07:33.715583 23046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "56e35bc7427f4c809d5abc9d8310d981" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0fa321ed1910400eb592b1a9c1348698"
I20260807 07:07:33.716079 23046 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fa321ed1910400eb592b1a9c1348698 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:07:33.715575 23084 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fa321ed1910400eb592b1a9c1348698 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:07:33.719516 23046 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0fa321ed1910400eb592b1a9c1348698 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e35bc7427f4c809d5abc9d8310d981 in term 1.
I20260807 07:07:33.720263 22784 leader_election.cc:302] T 00000000000000000000000000000000 P 56e35bc7427f4c809d5abc9d8310d981 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fa321ed1910400eb592b1a9c1348698, 56e35bc7427f4c809d5abc9d8310d981; no voters:
I20260807 07:07:33.720791 23101 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56e35bc7427f4c809d5abc9d8310d981 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:07:33.725014 23101 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56e35bc7427f4c809d5abc9d8310d981 [term 1 LEADER]: Becoming Leader. State: Replica: 56e35bc7427f4c809d5abc9d8310d981, State: Running, Role: LEADER
I20260807 07:07:33.725637 23101 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56e35bc7427f4c809d5abc9d8310d981 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e35bc7427f4c809d5abc9d8310d981" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 33025 } } peers { permanent_uuid: "a67718b839174e6aa7c280484be40a76" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 40539 } } peers { permanent_uuid: "0fa321ed1910400eb592b1a9c1348698" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 45545 } }
I20260807 07:07:33.740341 23116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56e35bc7427f4c809d5abc9d8310d981 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56e35bc7427f4c809d5abc9d8310d981. Latest consensus state: current_term: 1 leader_uuid: "56e35bc7427f4c809d5abc9d8310d981" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e35bc7427f4c809d5abc9d8310d981" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 33025 } } peers { permanent_uuid: "a67718b839174e6aa7c280484be40a76" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 40539 } } peers { permanent_uuid: "0fa321ed1910400eb592b1a9c1348698" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 45545 } } }
I20260807 07:07:33.740844 23116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56e35bc7427f4c809d5abc9d8310d981 [sys.catalog]: This master's current role is: LEADER
I20260807 07:07:33.743855 23129 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:07:33.753541 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:33.727721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39790 (local address 127.21.83.60:45545)
0807 07:07:33.727923 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:33.727929 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:33.727948 (+ 19us) server_negotiation.cc:407] Connection header received
0807 07:07:33.728097 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:33.728103 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:33.728184 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:33.728258 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:33.733538 (+ 5280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:33.734654 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:33.737180 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:33.737855 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:33.737952 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:33.744778 (+ 6826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:33.744787 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:33.744792 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:07:33.744843 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:33.748743 (+ 3900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:33.750191 (+ 1448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:33.750198 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:33.750202 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:33.750373 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:33.751241 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:33.751246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:33.751249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:33.751672 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:33.751941 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:33.752869 (+ 928us) server_negotiation.cc:299] Negotiation successful
0807 07:07:33.753170 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260807 07:07:33.760193 22792 tablet.cc:2369] T 00000000000000000000000000000000 P 56e35bc7427f4c809d5abc9d8310d981: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 07:07:33.760959 23129 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:07:33.777309 22974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:33.728509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52552 (local address 127.21.83.61:40539)
0807 07:07:33.728687 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:33.728692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:33.737192 (+ 8500us) server_negotiation.cc:407] Connection header received
0807 07:07:33.738186 (+ 994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:33.738192 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:33.738278 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:33.738350 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:33.746826 (+ 8476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:33.747964 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:33.751996 (+ 4032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:33.752704 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:33.752808 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:33.758772 (+ 5964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:33.758782 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:33.758788 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:07:33.758847 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:33.764946 (+ 6099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:33.772004 (+ 7058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:33.772014 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:33.772018 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:33.772274 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:33.773667 (+ 1393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:33.773674 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:33.773678 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:33.774193 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:33.774487 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:33.776545 (+ 2058us) server_negotiation.cc:299] Negotiation successful
0807 07:07:33.776914 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 07:07:33.779487 23046 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0fa321ed1910400eb592b1a9c1348698 [term 1 FOLLOWER]: Refusing update from remote peer 56e35bc7427f4c809d5abc9d8310d981: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:07:33.780702 23116 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 56e35bc7427f4c809d5abc9d8310d981 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fa321ed1910400eb592b1a9c1348698" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 45545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:07:33.782344 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:33.729548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38862 (local address 127.21.83.62:33025)
0807 07:07:33.729756 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:33.729761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:33.737867 (+ 8106us) server_negotiation.cc:407] Connection header received
0807 07:07:33.740924 (+ 3057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:33.740932 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:33.741039 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:33.741140 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:33.743451 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:33.744842 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:33.750021 (+ 5179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:33.750746 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:33.750862 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:33.766261 (+ 15399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:33.766269 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:33.766275 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:07:33.766334 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:33.770723 (+ 4389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:33.772855 (+ 2132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:33.772863 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:33.772867 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:33.773108 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:33.775052 (+ 1944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:33.775059 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:33.775063 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:33.775524 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:33.775841 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:33.780325 (+ 4484us) server_negotiation.cc:299] Negotiation successful
0807 07:07:33.780655 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 07:07:33.784293 23146 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260807 07:07:33.788862 22966 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.83.60" port: 45545 } attempt: 5
Aug 07 07:07:33 dist-test-slave-ncp9 krb5kdc[22742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786086452, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.83.61@KRBTEST.COM for kudu/127.21.83.60@KRBTEST.COM
I20260807 07:07:33.791057 22763 master_runner.cc:186] Error getting master registration for 127.21.83.61:40539: OK, instance_id { permanent_uuid: "a67718b839174e6aa7c280484be40a76" instance_seqno: 1786086453061644 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 07:07:33.804414 22763 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 07:07:33.810984 22972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:33.789476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.83.60:45545 (local address 127.0.0.1:39804)
0807 07:07:33.790095 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0807 07:07:33.790117 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 07:07:33.790254 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 07:07:33.790608 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 07:07:33.790613 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 07:07:33.791147 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0807 07:07:33.791363 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:07:33.791376 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:07:33.792819 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:07:33.792825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:07:33.794251 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 07:07:33.794267 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 07:07:33.796686 (+ 2419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 07:07:33.796692 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 07:07:33.796922 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:33.797648 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 07:07:33.797673 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 07:07:33.800475 (+ 2802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 07:07:33.806855 (+ 6380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:07:33.806861 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:07:33.806864 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:07:33.807420 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:07:33.808261 (+ 841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 07:07:33.808267 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 07:07:33.808271 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 07:07:33.808463 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 07:07:33.809839 (+ 1376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 07:07:33.809845 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 07:07:33.810033 (+ 188us) client_negotiation.cc:769] Sending connection context
0807 07:07:33.810175 (+ 142us) client_negotiation.cc:240] Negotiation successful
0807 07:07:33.810523 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484}
I20260807 07:07:33.813474 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:33.789914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39804 (local address 127.21.83.60:45545)
0807 07:07:33.790085 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:33.790091 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:33.790190 (+ 99us) server_negotiation.cc:407] Connection header received
0807 07:07:33.790389 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:33.790394 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:33.790475 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:33.790586 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:33.791540 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:33.792696 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:33.794404 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:33.795021 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:33.795142 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:33.800715 (+ 5573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 07:07:33.800722 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 07:07:33.800727 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 07:07:33.800769 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 07:07:33.806631 (+ 5862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:33.807707 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:33.807747 (+ 40us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:33.807752 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:33.808043 (+ 291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 07:07:33.808705 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 07:07:33.808713 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 07:07:33.808717 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 07:07:33.809606 (+ 889us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 07:07:33.811883 (+ 2277us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:33.812225 (+ 342us) server_negotiation.cc:299] Negotiation successful
0807 07:07:33.812752 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260807 07:07:33.819846 22966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a67718b839174e6aa7c280484be40a76: Bootstrap starting.
I20260807 07:07:33.822968 23114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fa321ed1910400eb592b1a9c1348698 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56e35bc7427f4c809d5abc9d8310d981. Latest consensus state: current_term: 1 leader_uuid: "56e35bc7427f4c809d5abc9d8310d981" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e35bc7427f4c809d5abc9d8310d981" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 33025 } } peers { permanent_uuid: "a67718b839174e6aa7c280484be40a76" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 40539 } } peers { permanent_uuid: "0fa321ed1910400eb592b1a9c1348698" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 45545 } } }
I20260807 07:07:33.823277 23114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fa321ed1910400eb592b1a9c1348698 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 07:07:33.823987 22966 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a67718b839174e6aa7c280484be40a76: Neither blocks nor log segments found. Creating new log.
I20260807 07:07:33.826560 22966 log.cc:824] T 00000000000000000000000000000000 P a67718b839174e6aa7c280484be40a76: Log is configured to *not* fsync() on all Append() calls
I20260807 07:07:33.829654 23151 mvcc.cc:205] Tried to move back new op lower bound from 7315810114665725952 to 7315810114466324480. Current Snapshot: MvccSnapshot[applied={T|T < 7315810114665725952}]
I20260807 07:07:33.834417 23114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fa321ed1910400eb592b1a9c1348698 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "56e35bc7427f4c809d5abc9d8310d981" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e35bc7427f4c809d5abc9d8310d981" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 33025 } } peers { permanent_uuid: "a67718b839174e6aa7c280484be40a76" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 40539 } } peers { permanent_uuid: "0fa321ed1910400eb592b1a9c1348698" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 45545 } } }
I20260807 07:07:33.834709 23114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fa321ed1910400eb592b1a9c1348698 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 07:07:33.834861 22966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a67718b839174e6aa7c280484be40a76: No bootstrap required, opened a new log
I20260807 07:07:33.848706 23162 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260807 07:07:33.849231 23101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56e35bc7427f4c809d5abc9d8310d981 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "56e35bc7427f4c809d5abc9d8310d981" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e35bc7427f4c809d5abc9d8310d981" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 33025 } } peers { permanent_uuid: "a67718b839174e6aa7c280484be40a76" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 40539 } } peers { permanent_uuid: "0fa321ed1910400eb592b1a9c1348698" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 45545 } } }
I20260807 07:07:33.849664 23101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56e35bc7427f4c809d5abc9d8310d981 [sys.catalog]: This master's current role is: LEADER
I20260807 07:07:33.853716 23129 catalog_manager.cc:1347] Generated new cluster ID: 938c8ef33fef4ef2ad1fd32397b4134f
I20260807 07:07:33.853849 23129 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:07:33.855414 22966 raft_consensus.cc:348] T 00000000000000000000000000000000 P a67718b839174e6aa7c280484be40a76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e35bc7427f4c809d5abc9d8310d981" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 33025 } } peers { permanent_uuid: "a67718b839174e6aa7c280484be40a76" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 40539 } } peers { permanent_uuid: "0fa321ed1910400eb592b1a9c1348698" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 45545 } }
I20260807 07:07:33.855890 22966 raft_consensus.cc:729] T 00000000000000000000000000000000 P a67718b839174e6aa7c280484be40a76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a67718b839174e6aa7c280484be40a76, State: Initialized, Role: FOLLOWER
I20260807 07:07:33.856434 22966 consensus_queue.cc:260] T 00000000000000000000000000000000 P a67718b839174e6aa7c280484be40a76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e35bc7427f4c809d5abc9d8310d981" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 33025 } } peers { permanent_uuid: "a67718b839174e6aa7c280484be40a76" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 40539 } } peers { permanent_uuid: "0fa321ed1910400eb592b1a9c1348698" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 45545 } }
I20260807 07:07:33.856377 22977 master_runner.cc:186] Error getting master registration for 127.21.83.61:40539: OK, instance_id { permanent_uuid: "a67718b839174e6aa7c280484be40a76" instance_seqno: 1786086453061644 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 07:07:33.871291 22977 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 07:07:33.872033 23163 sys_catalog.cc:455] T 00000000000000000000000000000000 P a67718b839174e6aa7c280484be40a76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e35bc7427f4c809d5abc9d8310d981" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 33025 } } peers { permanent_uuid: "a67718b839174e6aa7c280484be40a76" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 40539 } } peers { permanent_uuid: "0fa321ed1910400eb592b1a9c1348698" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 45545 } } }
I20260807 07:07:33.872512 23163 sys_catalog.cc:458] T 00000000000000000000000000000000 P a67718b839174e6aa7c280484be40a76 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 07:07:33.873591 22966 sys_catalog.cc:564] T 00000000000000000000000000000000 P a67718b839174e6aa7c280484be40a76 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:07:33.873667 22931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "56e35bc7427f4c809d5abc9d8310d981" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a67718b839174e6aa7c280484be40a76"
I20260807 07:07:33.874231 22931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a67718b839174e6aa7c280484be40a76 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:07:33.877758 22931 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a67718b839174e6aa7c280484be40a76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e35bc7427f4c809d5abc9d8310d981 in term 1.
I20260807 07:07:33.882063 22928 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a67718b839174e6aa7c280484be40a76 [term 1 FOLLOWER]: Refusing update from remote peer 56e35bc7427f4c809d5abc9d8310d981: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 07:07:33.882433 23129 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:07:33.884327 23129 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:07:33.884819 23116 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 56e35bc7427f4c809d5abc9d8310d981 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a67718b839174e6aa7c280484be40a76" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 40539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:07:33.910030 23129 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56e35bc7427f4c809d5abc9d8310d981: Generated new TSK 0
I20260807 07:07:33.911680 23129 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 07:07:33.922611 23179 sys_catalog.cc:455] T 00000000000000000000000000000000 P a67718b839174e6aa7c280484be40a76 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "56e35bc7427f4c809d5abc9d8310d981" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e35bc7427f4c809d5abc9d8310d981" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 33025 } } peers { permanent_uuid: "a67718b839174e6aa7c280484be40a76" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 40539 } } peers { permanent_uuid: "0fa321ed1910400eb592b1a9c1348698" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 45545 } } }
I20260807 07:07:33.923105 23179 sys_catalog.cc:458] T 00000000000000000000000000000000 P a67718b839174e6aa7c280484be40a76 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 07:07:33.923029 23163 sys_catalog.cc:455] T 00000000000000000000000000000000 P a67718b839174e6aa7c280484be40a76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56e35bc7427f4c809d5abc9d8310d981. Latest consensus state: current_term: 1 leader_uuid: "56e35bc7427f4c809d5abc9d8310d981" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e35bc7427f4c809d5abc9d8310d981" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 33025 } } peers { permanent_uuid: "a67718b839174e6aa7c280484be40a76" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 40539 } } peers { permanent_uuid: "0fa321ed1910400eb592b1a9c1348698" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 45545 } } }
I20260807 07:07:33.923306 23163 sys_catalog.cc:458] T 00000000000000000000000000000000 P a67718b839174e6aa7c280484be40a76 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 07:07:33.924268 23101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56e35bc7427f4c809d5abc9d8310d981 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "56e35bc7427f4c809d5abc9d8310d981" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e35bc7427f4c809d5abc9d8310d981" member_type: VOTER last_known_addr { host: "127.21.83.62" port: 33025 } } peers { permanent_uuid: "a67718b839174e6aa7c280484be40a76" member_type: VOTER last_known_addr { host: "127.21.83.61" port: 40539 } } peers { permanent_uuid: "0fa321ed1910400eb592b1a9c1348698" member_type: VOTER last_known_addr { host: "127.21.83.60" port: 45545 } } }
I20260807 07:07:33.924566 23101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56e35bc7427f4c809d5abc9d8310d981 [sys.catalog]: This master's current role is: LEADER
I20260807 07:07:33.935709 23190 catalog_manager.cc:1259] Loaded cluster ID: 938c8ef33fef4ef2ad1fd32397b4134f
I20260807 07:07:33.935829 23190 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a67718b839174e6aa7c280484be40a76: loading cluster ID for follower catalog manager: success
I20260807 07:07:33.939581 23190 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a67718b839174e6aa7c280484be40a76: acquiring CA information for follower catalog manager: success
I20260807 07:07:33.947885 23190 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a67718b839174e6aa7c280484be40a76: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260807 07:07:34.019894 23095 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:07:34.010448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39808 (local address 127.21.83.60:45545)
0807 07:07:34.010629 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:34.010635 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:34.017605 (+ 6970us) server_negotiation.cc:407] Connection header received
0807 07:07:34.017949 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:34.017956 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:34.018062 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:34.018138 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:34.019620 (+ 1482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39808: BlockingRecv error: recv got EOF from 127.0.0.1:39808 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260807 07:07:34.025311 22974 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:07:34.010286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52562 (local address 127.21.83.61:40539)
0807 07:07:34.010531 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:34.010538 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:34.023051 (+ 12513us) server_negotiation.cc:407] Connection header received
0807 07:07:34.023526 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:34.023533 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:34.023638 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:34.023719 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:34.025045 (+ 1326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52562: BlockingRecv error: recv got EOF from 127.0.0.1:52562 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260807 07:07:34.026921 23090 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:07:34.017685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38864 (local address 127.21.83.62:33025)
0807 07:07:34.017878 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:34.017884 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:34.025283 (+ 7399us) server_negotiation.cc:407] Connection header received
0807 07:07:34.025516 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:34.025523 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:34.025606 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:34.025675 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:34.026651 (+ 976us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38864: BlockingRecv error: recv got EOF from 127.0.0.1:38864 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 07:07:34.044801 22974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:34.034988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52574 (local address 127.21.83.61:40539)
0807 07:07:34.035154 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:34.035160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:34.037418 (+ 2258us) server_negotiation.cc:407] Connection header received
0807 07:07:34.037645 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:34.037651 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:34.037881 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:34.038065 (+ 184us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:07:34.038537 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:34.039687 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:34.041711 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:34.042417 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:34.042537 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:34.043254 (+ 717us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:07:34.043964 (+ 710us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:07:34.044092 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:34.044344 (+ 252us) server_negotiation.cc:299] Negotiation successful
0807 07:07:34.044438 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260807 07:07:34.054860 23095 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:07:34.051936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39816 (local address 127.21.83.60:45545)
0807 07:07:34.052118 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:34.052124 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:34.053417 (+ 1293us) server_negotiation.cc:407] Connection header received
0807 07:07:34.053588 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:34.053593 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:34.053718 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:34.053824 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:34.054635 (+ 811us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39816: BlockingRecv error: recv got EOF from 127.0.0.1:39816 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260807 07:07:34.067808 22974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:34.059131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52580 (local address 127.21.83.61:40539)
0807 07:07:34.059311 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:34.059317 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:34.061497 (+ 2180us) server_negotiation.cc:407] Connection header received
0807 07:07:34.061689 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:34.061695 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:34.061806 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:34.061906 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:07:34.062382 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:34.063536 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:34.065558 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:34.066234 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:34.066367 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:34.066912 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:07:34.067102 (+ 190us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:07:34.067173 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:34.067397 (+ 224us) server_negotiation.cc:299] Negotiation successful
0807 07:07:34.067488 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260807 07:07:34.105489 23095 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:07:34.103258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39822 (local address 127.21.83.60:45545)
0807 07:07:34.103466 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:34.103473 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:34.103882 (+ 409us) server_negotiation.cc:407] Connection header received
0807 07:07:34.104075 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:34.104080 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:34.104174 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:34.104275 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:34.105243 (+ 968us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39822: BlockingRecv error: recv got EOF from 127.0.0.1:39822 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 07:07:34.122741 22974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:34.114876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52590 (local address 127.21.83.61:40539)
0807 07:07:34.115068 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:34.115074 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:34.116611 (+ 1537us) server_negotiation.cc:407] Connection header received
0807 07:07:34.116808 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:34.116814 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:34.116927 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:34.117043 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:07:34.117512 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:34.118730 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:34.120628 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:34.121341 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:34.121461 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:34.121944 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:07:34.122085 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:07:34.122171 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:34.122323 (+ 152us) server_negotiation.cc:299] Negotiation successful
0807 07:07:34.122411 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260807 07:07:34.131359 23095 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:07:34.129243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39828 (local address 127.21.83.60:45545)
0807 07:07:34.129418 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:34.129424 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:34.129925 (+ 501us) server_negotiation.cc:407] Connection header received
0807 07:07:34.130082 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:34.130087 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:34.130180 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:34.130286 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:34.131125 (+ 839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39828: BlockingRecv error: recv got EOF from 127.0.0.1:39828 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 07:07:34.145040 22974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:34.138366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52594 (local address 127.21.83.61:40539)
0807 07:07:34.138538 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:34.138544 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:34.139077 (+ 533us) server_negotiation.cc:407] Connection header received
0807 07:07:34.139266 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:34.139272 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:34.139372 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:34.139464 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:07:34.139947 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:34.141119 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:34.143005 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:34.143708 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:34.143826 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:34.144276 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:07:34.144402 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:07:34.144490 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:34.144631 (+ 141us) server_negotiation.cc:299] Negotiation successful
0807 07:07:34.144710 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260807 07:07:34.157429 23095 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:07:34.154758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39834 (local address 127.21.83.60:45545)
0807 07:07:34.155052 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:34.155058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:34.155897 (+ 839us) server_negotiation.cc:407] Connection header received
0807 07:07:34.156082 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:34.156088 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:34.156182 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:34.156289 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:34.157213 (+ 924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39834: BlockingRecv error: recv got EOF from 127.0.0.1:39834 (error 108)
Metrics: {"server-negotiator.queue_time_us":171}
I20260807 07:07:34.179458 22974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:34.171156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52608 (local address 127.21.83.61:40539)
0807 07:07:34.171345 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:34.171352 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:34.171940 (+ 588us) server_negotiation.cc:407] Connection header received
0807 07:07:34.172159 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:34.172170 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:34.172272 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:34.172386 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:07:34.172865 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:34.174158 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:34.176316 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:34.177094 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:34.177248 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:34.178053 (+ 805us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:07:34.178242 (+ 189us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:07:34.178352 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:34.178962 (+ 610us) server_negotiation.cc:299] Negotiation successful
0807 07:07:34.179067 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260807 07:07:34.199302 23095 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:07:34.196020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39842 (local address 127.21.83.60:45545)
0807 07:07:34.196286 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:34.196293 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:34.197556 (+ 1263us) server_negotiation.cc:407] Connection header received
0807 07:07:34.197787 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:34.197793 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:34.197892 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:34.198013 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:34.199038 (+ 1025us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39842: BlockingRecv error: recv got EOF from 127.0.0.1:39842 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
I20260807 07:07:34.227237 22974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:34.220058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52622 (local address 127.21.83.61:40539)
0807 07:07:34.220264 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:34.220270 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:34.220866 (+ 596us) server_negotiation.cc:407] Connection header received
0807 07:07:34.221121 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:34.221131 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:34.221225 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:34.221331 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:07:34.221846 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:34.223032 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:34.225076 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:34.225736 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:34.225858 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:34.226391 (+ 533us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:07:34.226602 (+ 211us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:07:34.226697 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:34.226861 (+ 164us) server_negotiation.cc:299] Negotiation successful
0807 07:07:34.226950 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260807 07:07:34.241286 23095 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:07:34.238858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39846 (local address 127.21.83.60:45545)
0807 07:07:34.239062 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:34.239069 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:34.239630 (+ 561us) server_negotiation.cc:407] Connection header received
0807 07:07:34.239844 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:34.239849 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:34.239943 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:34.240064 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:34.241038 (+ 974us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39846: BlockingRecv error: recv got EOF from 127.0.0.1:39846 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260807 07:07:34.284631 22974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:34.277935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52638 (local address 127.21.83.61:40539)
0807 07:07:34.278177 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:34.278183 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:34.279868 (+ 1685us) server_negotiation.cc:407] Connection header received
0807 07:07:34.280090 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:34.280095 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:34.280186 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:34.280269 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:07:34.280705 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:34.281497 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:34.282813 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:34.283317 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:34.283430 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:34.283860 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:07:34.283994 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:07:34.284081 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:34.284234 (+ 153us) server_negotiation.cc:299] Negotiation successful
0807 07:07:34.284315 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260807 07:07:34.292446 23095 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:07:34.290376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39854 (local address 127.21.83.60:45545)
0807 07:07:34.290589 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:34.290595 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:34.291068 (+ 473us) server_negotiation.cc:407] Connection header received
0807 07:07:34.291249 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:34.291256 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:34.291348 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:34.291429 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:34.292212 (+ 783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39854: BlockingRecv error: recv got EOF from 127.0.0.1:39854 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 07:07:34.365579 22974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:34.359621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52648 (local address 127.21.83.61:40539)
0807 07:07:34.359811 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:34.359817 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:34.360191 (+ 374us) server_negotiation.cc:407] Connection header received
0807 07:07:34.360362 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:34.360368 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:34.360457 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:34.360565 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:07:34.360933 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:34.362095 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:34.363549 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:34.364240 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:34.364369 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:34.364790 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:07:34.364989 (+ 199us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:07:34.365066 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:34.365196 (+ 130us) server_negotiation.cc:299] Negotiation successful
0807 07:07:34.365285 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260807 07:07:34.373548 23095 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:07:34.371562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39864 (local address 127.21.83.60:45545)
0807 07:07:34.371757 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:34.371762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:34.372096 (+ 334us) server_negotiation.cc:407] Connection header received
0807 07:07:34.372276 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:34.372281 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:34.372372 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:34.372478 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:34.373282 (+ 804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39864: BlockingRecv error: recv got EOF from 127.0.0.1:39864 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 07:07:34.509835 22974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:34.504854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52656 (local address 127.21.83.61:40539)
0807 07:07:34.505101 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:34.505108 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:34.505347 (+ 239us) server_negotiation.cc:407] Connection header received
0807 07:07:34.505562 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:34.505586 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:34.505669 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:34.505750 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:07:34.506164 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:34.506851 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:34.508172 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:34.508568 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:34.508684 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:34.509105 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:07:34.509210 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:07:34.509285 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:34.509426 (+ 141us) server_negotiation.cc:299] Negotiation successful
0807 07:07:34.509508 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260807 07:07:34.517143 23095 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:07:34.515344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39870 (local address 127.21.83.60:45545)
0807 07:07:34.515550 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:34.515555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:34.515858 (+ 303us) server_negotiation.cc:407] Connection header received
0807 07:07:34.516013 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:34.516017 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:34.516091 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:34.516176 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:34.516886 (+ 710us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39870: BlockingRecv error: recv got EOF from 127.0.0.1:39870 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260807 07:07:34.781843 22974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:34.776987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52666 (local address 127.21.83.61:40539)
0807 07:07:34.777169 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:34.777173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:34.777384 (+ 211us) server_negotiation.cc:407] Connection header received
0807 07:07:34.777601 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:34.777605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:34.777700 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:34.777785 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:07:34.778135 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:34.778872 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:34.780328 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:34.780742 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:34.780840 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:34.781297 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:07:34.781410 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:07:34.781474 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:34.781571 (+ 97us) server_negotiation.cc:299] Negotiation successful
0807 07:07:34.781639 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260807 07:07:34.789767 23095 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 07:07:34.787848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39876 (local address 127.21.83.60:45545)
0807 07:07:34.788064 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:34.788071 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:34.788244 (+ 173us) server_negotiation.cc:407] Connection header received
0807 07:07:34.788459 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:34.788465 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:34.788561 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:34.788648 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0807 07:07:34.789523 (+ 875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39876: BlockingRecv error: recv got EOF from 127.0.0.1:39876 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 07:07:34.853989 23162 catalog_manager.cc:1259] Loaded cluster ID: 938c8ef33fef4ef2ad1fd32397b4134f
I20260807 07:07:34.854120 23162 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0fa321ed1910400eb592b1a9c1348698: loading cluster ID for follower catalog manager: success
I20260807 07:07:34.857017 23162 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0fa321ed1910400eb592b1a9c1348698: acquiring CA information for follower catalog manager: success
I20260807 07:07:34.859297 23162 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0fa321ed1910400eb592b1a9c1348698: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 07:07:35.310881 23368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:35.305696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52680 (local address 127.21.83.61:40539)
0807 07:07:35.306117 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:35.306122 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:35.306280 (+ 158us) server_negotiation.cc:407] Connection header received
0807 07:07:35.306453 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:35.306457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:35.306535 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:35.306633 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:07:35.307054 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:35.307914 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:35.309350 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:35.309808 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:35.309910 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:35.310258 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:07:35.310357 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:07:35.310419 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:35.310507 (+ 88us) server_negotiation.cc:299] Negotiation successful
0807 07:07:35.310591 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":223,"threads_started":1}
I20260807 07:07:35.321359 23378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:35.316056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39878 (local address 127.21.83.60:45545)
0807 07:07:35.316617 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:35.316622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:35.316637 (+ 15us) server_negotiation.cc:407] Connection header received
0807 07:07:35.316694 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:35.316697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:35.316843 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:35.316962 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:07:35.317320 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:35.318093 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:35.319374 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:35.319787 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:35.319874 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:35.320286 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:07:35.320792 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:07:35.320929 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:35.321018 (+ 89us) server_negotiation.cc:299] Negotiation successful
0807 07:07:35.321083 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":336,"threads_started":1}
I20260807 07:07:35.344874 23368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:35.327287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52684 (local address 127.21.83.61:40539)
0807 07:07:35.327638 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:35.327644 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:35.339055 (+ 11411us) server_negotiation.cc:407] Connection header received
0807 07:07:35.339356 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:35.339361 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:35.339453 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:35.339516 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:07:35.340311 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:35.341155 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:35.343000 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:35.343375 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:35.343484 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:35.344062 (+ 578us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:07:35.344201 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:07:35.344288 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:35.344502 (+ 214us) server_negotiation.cc:299] Negotiation successful
0807 07:07:35.344592 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231}
I20260807 07:07:35.345402 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:35.326658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38870 (local address 127.21.83.62:33025)
0807 07:07:35.338552 (+ 11894us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:35.338559 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:35.338584 (+ 25us) server_negotiation.cc:407] Connection header received
0807 07:07:35.338737 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:35.338744 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:35.338970 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:35.339208 (+ 238us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:07:35.339814 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:35.340839 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:35.343038 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:35.343604 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:35.343723 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:35.344230 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:07:35.344653 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:07:35.344759 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:35.345003 (+ 244us) server_negotiation.cc:299] Negotiation successful
0807 07:07:35.345100 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11750,"thread_start_us":11644,"threads_started":1}
I20260807 07:07:35.345525 23378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 07:07:35.327091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39882 (local address 127.21.83.60:45545)
0807 07:07:35.327495 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0807 07:07:35.327501 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 07:07:35.339303 (+ 11802us) server_negotiation.cc:407] Connection header received
0807 07:07:35.339575 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 07:07:35.339581 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 07:07:35.339682 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 07:07:35.339765 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 07:07:35.340504 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:35.341425 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:35.343465 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 07:07:35.344093 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 07:07:35.344212 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 07:07:35.344718 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 07:07:35.344847 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 07:07:35.344933 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0807 07:07:35.345154 (+ 221us) server_negotiation.cc:299] Negotiation successful
0807 07:07:35.345249 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266}
I20260807 07:07:35.351541 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 22763
I20260807 07:07:35.353061 22852 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:07:35.467444 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 22763
I20260807 07:07:35.493621 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 22868
I20260807 07:07:35.494768 22960 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:07:35.604038 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 22868
I20260807 07:07:35.628450 21836 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu pid 22977
I20260807 07:07:35.629488 23076 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 07:07:35.735119 21836 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyUa4Vb/build/asan/bin/kudu with pid 22977
2026-08-07T07:07:35Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5274 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (10503 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (377625 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST